RESPONSE TO REQUIREMENT FOR INFORMATION Attorney Docket No.: Q77095 

Application No.: 10/646,753 



REMARKS 

In the Amendment filed April 11, 2008, Applicant indicated that no Office Action had 
been received for Japanese Application No. 2002-1 85874. However, in the Information 
Disclosure Statement filed December 4, 2007, multiple references were listed that were cited in 
an Office Action for Japanese Application No. 2002- 1 85874 dated September 1 1 , 2007 (along 
with references for Japanese Application Nos. 2002-249212 and 2002-249213). Applicant 
submits that the erroneous statement provided in the Amendment filed April 11, 2008 was 
unintentionally submitted. 

In response to the Requirement for Information dated June 2, 2008, Applicant responds to 
the Examiner's inquiries as follows. 

1. Yes 

2. Copies of the Notice of Grounds for Rejection issued September 1 1, 2007, in both 
Japanese and English languages, are attached hereto. 

3. Yes 

4. Copies of the Argument and Amendment submitted November 8, 2007, in both 
Japanese and English languages, are attached hereto. 

5. Yes 

6. Copies of the Decision of Rejection issued December 4, 2007, in both Japanese and 
English languages, are attached hereto. 

7. Yes 
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RESPONSE TO REQUIREMENT FOR INFORMATION Attorney Docket No.: Q77095 

Application No.: 10/646,753 

8. Copies of the Amendment submitted February 4, 2008, in both Japanese and English 
languages, are attached hereto. 1 

9. Copies of the Notice of Grounds for Rejection issued September 4, 2007, Argument 
and Amendment submitted November 5, 2007, Decision of Rejection issued March 18, 2008, 
and Appeal Brief and Amendment submitted May 13, 2008, in both Japanese and English 
languages, are attached hereto. 2 

10. Copies of the Notice of Grounds for Rejection issued September 4, 2007, and the 
Argument and Amendment submitted November 5, 2007, in both Japanese and English 
languages, are attached hereto.- 

1 1 . Copies of the Notice of Grounds for Rejection issued September 4, 2007, and the 
Argument and Amendment submitted November 5, 2007, in both Japanese and English 
languages, are attached hereto. 4 

12. Copies of the Notice of Grounds for Rejection issued April 24, 2007,- Argument and 
Amendment submitted June 25, 2007, Decision of Rejection issued October 16, 2007, and the 



1 With further regard to Japanese Application JP 2002-185874, Applicant will be submitting English 
and Japanese versions of an Appeal Brief submitted February 13, 2008 and a Decision of Grant issued 
on March 18, 2008, in a supplemental response. 

1 Japanese Application JP 2002-249212 is currently in appeal proceedings. 

2 With further regard to Japanese Application JP 2002-249213, Applicant will be submitting English 
and Japanese versions of a Decision of Grant issued March 25, 2008, in a supplemental response. 

4 With further regard to Japanese Application JP 2002-284126, Applicant will be submitting English 
and Japanese versions of a Decision of Grant issued March 25, 2008, in a supplemental response. 

5 Applicant notes that the English translation of the Notice of Grounds for Rejection issued April 24, 
2007 contains a typographical error in the Application Number. The translation indicates the 
Application Number as 2002-284128, however, the Application Number should be 2002-284127. 
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RESPONSE TO REQUIREMENT FOR INFORMATION Attorney Docket No.: Q77095 

Application No.: 10/646,753 

Appeal Brief and Amendment submitted December 14, 2007, in both Japanese and English 
languages, are attached hereto.- 

13. Copies of the Notice of Grounds for Rejection issued April 24, 2007 and the 
Decision of Rejection issued October 16, 2007, in both Japanese and English languages are 
attached hereto. 2 

14. Applicant submits that there are no applications, other than the instant application, 
that claim priority from any of the following Japanese application numbers: JP 2002-185874, 
2002-249212, 2002-249213, 2002-284126, 2002-284127 and 2002 - 284128. 

In view of the above, reconsideration and allowance of this application are now believed 
to be in order, and such actions are hereby solicited. If any points remain in issue which the 
Examiner feels may be best resolved through a personal or telephone interview, the Examiner is 
kindly requested to contact the undersigned at the telephone number listed below. 



* Japanese Application JP 2002-284127 is currently in appeal proceedings. 

With further regard to Japanese Application JP 2002-284128, Applicant will be submitting English 
and Japanese versions of an Argument and Amendment submitted June 25, 2007 and an Appeal Brief 
and Amendment submitted December 14, 2008, in a supplemental response. Japanese Application JP 
2002-284127 is currently in appeal proceedings. 
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RESPONSE TO REQUIREMENT FOR INFORMATION 
Application No.: 10/646,753 



Attorney Docket No.: Q77095 



The USPTO is directed and authorized to charge all required fees, except for the Issue 
Fee and the Publication Fee, to Deposit Account No. 19-4880. Please also credit any 
overpayments to said Deposit Account. 

Respectfully submitted, 



SUGHRUE MION, PLLC 
Telephone: (202) 293-7060 
Facsimile: (202) 293-7860 

WASHINGTON OFFICE 

23373 

CUSTOMER NUMBER 

Date: August 4, 2008 
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File No: P26913J Issue No: 447784 Issue Date: September 11, 2007 



Notice of Grounds for Rejection 



Application Number: 
5 Drafting Date: 



2002-185874 



September 3, 2 007 



Examiner : 



Norikazu Arai 



8937 5H00 



Applicant ' s Representative : 
Applied Statutes: 



Masashi Yanagida et al . 
Article 29, Paragraph 1; Article 29, 
Paragraph 2 
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The present application is rejected based on the following Grounds. 
In the case that the applicant has comments regarding the rejection, 
please file an Argument within 60 days of the issue date of this 
Notice . 



[Grounds 1, 2] 

1. The inventions defined in the following claims of the present 

2 0 application are the same as the inventions disclosed in the following 

publications , which were distributed either domestically or overseas , 
or inventions that were made available to the public via 
telecommunications lines, prior to the filing date of the present 
application. Therefore, patent rights cannot be granted, according 
25 to the regulations prescribed in Article 29, Paragraph 1, Number 3 
of the Patent Law. 

2 . The inventions defined in the following claims of the present 
application could have been made easily by those having ordinary skill 

3 0 in the art, based on the inventions disclosed in the following 

publications , which were distributed either domestically or overseas , 
or inventions that were made available to the public via 
telecommunications lines, prior to the filing date of the present 
application. Therefore, patent rights cannot be granted, according 
3 5 to the regulations prescribed in Article 29, Paragraph 2 of the Patent 
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GROUNDS 



Law. 



NOTES (please refer to the List of Cited References) 



5 • Claims: 1, 4, and 7 

• Grounds: 1 and 2 

• Cited Reference: 1 

• Remarks : 



10 The "degree of matching" disclosed in Claims 1, 2, 7, 8, 13, and 14, 
as well as paragraph [0098] of Cited Reference 1 correspond to the 
"similarity" recited in Claims 1,4, and 7 of the present application. 
Accordingly, the inventions defined in Claims 1, 4, and 7 of the 
present application are disclosed in Cited Reference 1 . In addition, 

15 the inventions defined in Claims 1,4, and 7 of the present application 
would have been easily achieved by those skilled in the art, based 
on the disclosure of Cited Reference 1. 



• Claims : 2,5, and 8 
2 0 • Ground: 2 

• Cited References: 1 and 2 

• Remarks : 



Dividing frames into pluralities of regions and then synthesizing 
25 the frames based on similarities was a well known technique prior 
to the filing date of the present application (refer to Claim 1 and 
paragraphs [0035] through [0042] of Cited Reference 2) 

Accordingly, it would have been easy for those skilled in the art 
3 0 to apply the well known technique described above to the invention 
disclosed in Cited Reference 1, to achieve the inventions recited 
in Claims 2, 5, and 8 of the present application. 



• Claims: 3, 6, and 9 

35 • Ground: 2 



• Cited References: 1 and 3 

• Remarks : 

Weighting and synthesizing each of a plurality of subjects 
5 individually was a well known technique prior to the filing date of 
the present application (refer to the abstract of Cited Reference 
3) . 

Accordingly, it would have been easy for those skilled in the art 
10 to apply the well known technique described above to the invention 
disclosed in Cited Reference 1, to achieve the inventions recited 
in Claims 3, 6, and 9 of the present application. 

If new Grounds for Rejection are found, the applicant will be 
15 notified. 

List of Cited References 

1. Japanese Unexamined Patent Publication No. 2001-177714 
20 2. Japanese Unexamined Patent Publication No. 2000-354244 
3. International Publication No. 00/008860 

Record of Prior Art Document Search Results 

25 • Field of Search: IPC G06T 7/20 

Database Name 

The Record of Prior Art Document Search Results does not constitute 
the Grounds for Rejection. 

30 
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File No. : Application No. 2002-185874 



Submitted: 11/8/2007 



Amendment 

Commissioner of the Patent Office 



2002-185874 



[Name of Document] 
[Addressee] 
[Indication of Case] 

[Application Number] 
[Applicant] 

[ID Number] 

[Name] 
[Representative] 

[ID Number] 

[Patent Attorney] 

[Name] 
[Submission Number] 
[Amendment 1] 

[Document to be Amended] Specification 

[Item to be Amended] 

[Manner of Amendment] 

[Contents of Amendment] 
[Scope of Demand for Patent] 
[Claim 1] 

An image processing method, for acquiring a processed frame 
by performing image processing on a desired frame sampled from a video 
image, said method comprising the steps of: 

dividing the desired frame into a plurality of subject regions 
which are included within the desired frame; 

for each of the 



306037311 
FUJIFILM Corp. 



100073184 



Masashi Yanagida 
447784 



Scope of Demand for Patent 
Change 



computing < 



ifey similarities bctwcc: 



plurality of subject regions of said desired frame and corresponding 
subject regions that correspond to the subject regions within at least 
one frame which is temporally before and after said desired frame; 
3 0 and 

acquiring said processed frame by obtaining a weighting 
coefficients that becomes greater if- as said similarity similarities 
becomes greater, then weighting the corresponding subject regions 
of said at least one frame with said weighting coefficients, and 
35 synthesizing said weighted frame and said desired frame. 



[Claim 2] 

An image processor for acquiring a processed frame by- 
performing image processing on a desired frame sampled from a video 
image, said image processor comprising: 
5 similarity computation means for dividing the desired frame 

into a plurality of subject regions which are included within the 
desired frame, and computing a similarity similarities between for 
each of the plurality of subject regions of said desired frame and 
corresponding subject regions that correspond to the subject regions 

10 within at least one frame which is temporally before and after said 
desired frame; and 

synthesis means for acquiring said processed frame by 
obtaining a weighting coefficients that become© greater i# as said 
similarity similarities become-s greater, then weighting the 

15 corresponding subject regions of said at least one frame with said 
weighting coefficients, and synthesizing said weighted frame and said 
desired frame. 
[Claim 3] 

A program for causing a computer to execute an image processing 

2 0 method of acquiring a processed frame by performing image processing 

on a desired frame sampled from a video image, said program 
comprising: 

a similarity computation procedure of dividing the desired 
frame into a plurality of subject regions which are included within 
25 the desired frame, and computing a similarity similarities between 
for each of the plurality of subject regions of said desired frame 
and corresponding subject regions that correspond to the subject 
regions within at least one frame which is temporally before and after 
said desired frame; and 

3 0 synthesis means for acquiring said processed frame by 

obtaining a weighting coefficients that become-s greater if- as said 
similarity similarities become-s- greater, then weighting the 
corresponding subject regions of said at least one frame with said 
weighting coefficients, and synthesizing said weighted frame and said 
3 5 desired frame. 



[Amendment 2 ] 

[Document to be Amended] Specification 
[Item to be Amended] Paragraph 0044 

[Manner of Amendment] Change 
[Contents of Amendment] 

[0044] 

As illustrated in Fig. 9, the motion-vector computation means 
5 partitions frame Fr x into m X n block-shaped areas Al(m, n) and 
moves each of the areas A(m, n) in parallel with frame Fro. . And when 
a correlation between pixel values in area Al (m, n) and frame Fr 2 
is highest, the moved quantity and moving direction of area Al(m, 
n) is computed as motion vector V0 (m, n) for that area Al (m, n) . Note 
that when the accumulation of the squares of differences between pixel 
values of area Al(m, n) and frame Fr 2 or accumulation of absolute 
values is smallest, the correlation is judged to be highest. 
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B 13J Issue Number: 614961 Issue Date: 12/4/2007 
NOTICE OF DECISION OF REJECTION 



Patent Application Number: 2002-185874 

Drafting Date: November 27, 2007 

Patent Office Examiner: Norikazu Arai 8937 5H00 

Title of invention: Method, Apparatus, and Program for 

Image Processing 

Applicant: FUJIFILM Corp. 

Applicant's Representative: Masashi Yanagida et al . 

The present application is to be rejected based on the grounds 
described in the Notice of Grounds for Rejection, dated September 
3, 2007. 

Note that the contents of the Argument and the Amendment were 
considered, but no basis sufficient to overturn the Grounds for 
Rejection was found. 

Remarks 

The applicant contended that "that is, the Examiner asserts that Cited 
Reference 3 discloses weighting and synthesizing each of a plurality 
of subjects individually, in the Notice of Grounds for Rejection. 
However, Cited Reference 3 merely discloses weighting each of a 
plurality of unit regions, and does not disclose nor suggest weighting 
each of a plurality of subject regions individually". 

However, the "sprites" disclosed in Cited Reference 3 and the "subject 
regions" recited in Claims 1 through 3 of the present application 
correspond to each other in that they are both regions that include 
objects. In addition, dividing into regions was also well known 
prior to the filing date of the present application. Therefore, it 
would have been easy for those skilled in the art to apply the well 
known technique described above to the invention disclosed in Cited 
Reference 1, to achieve the inventions recited in Claims 1 through 
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File No. 



Application No. 2002-185874 



Submitted: 2/4/2008 



Amendment 

Commissioner of the Patent Office 



2008-34 
2002-185874 



306037311 
FUJIFILM Corp. 



100073184 



Masashi Yanagida 



Scope of Demand for Patent 
Change 



[Name of Document] 
[Addressee] 
[Indication of Case] 

[Appeal Number] 

[Application Number] 
[Applicant] 

[ID Number] 

[Name] 
[Representative] 

[ID Number] 

[Patent Attorney] 

[Name] 
[Amendment 1] 

[Document to be Amended] Specification 

[Item to be Amended] 

[Manner of Amendment] 

[Contents of Amendment] 
[Scope of Demand for Patent] 
[Claim 1] 

An image processing method, for acquiring a processed frame 
by performing image processing on a desired frame sampled from a video 
image, said method comprising the steps of: 

dividing the desired frame into a plurality of regions, then 
organizing the plurality of regions into regions according to 
movement therein, to divide the desired frame into a plurality of 
subject regions which are included within the desired frame; 

computing similarities for each of the plurality of subject 
regions of said desired frame and corresponding subject regions that 
correspond to the subject regions within at least one frame which 
is temporally before and after said desired frame; and 

acquiring said processed frame by obtaining a weighting 
coefficients that becomes greater as said similarities becomes 
greater, then weighting the corresponding subject regions of said 
at least one frame with said weighting coefficients, and synthesizing 



said weighted frame and said desired frame. 
[Claim 2] 

An image processor for acquiring a processed frame by 
5 performing image processing on a desired frame sampled from a video 
image, said image processor comprising: 

similarity computation means for dividing the desired frame 
into a plurality of regions, then organizing the plurality of regions 
into regions according to movement therein, to divide the desired 
10 frame into a plurality of subject regions which are included within 
the desired frame, and computing similarities for each of the 
plurality of subject regions of said desired frame and corresponding 
subject regions that correspond to the subject regions within at least 
one frame which is temporally before and after said desired frame; 
15 and 

synthesis means for acquiring said processed frame by 
obtaining a weighting coefficients that becomes greater as said 
similarities becomes greater, weighting the corresponding subject 
regions of said at least one frame with said weighting coefficients, 
20 and synthesizing said weighted frame and said desired frame. 

[Claim 3] 

A program for causing a computer to execute an image processing 
method of acquiring a processed frame by performing image processing 
25 on a desired frame sampled from a video image, said program 
comprising: 

a similarity computation procedure of dividing the desired 
frame into a plurality of regions, then organizing the plurality of 
regions into regions according to movement therein, to divide the 

3 0 desired frame into a plurality of subject regions which are included 
within the desired frame, and computing similarities for each of the 
plurality of subject regions of said desired frame and corresponding 
subject regions that correspond to the subject regions within at least 
one frame which is temporally before and after said desired frame; 

3 5 and 



synthesis means for acquiring said processed frame by 
obtaining a weighting coefficients that becomes greater as said 
similarity similarities becomes greater, weighting the 
corresponding subject regions of said at least one frame with said 
5 weighting coefficients, and synthesizing said weighted frame and said 
desired frame . 
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File No: P27179J Issue No: 435087 Issue Date: September 4, 2007 



Notice of Grounds for Rejection 

Application Number: 2002-249212 

5 Drafting Date: August 28, 2008 

Examiner: Michiharu Sugawara 8725 5C00 

Applicant's Representative: Masashi Yanagida et al . 

Applied Statutes: Article 29, Paragraph 2; Article 36 

10 The present application is rejected based on the following Grounds. 
In the case that the applicant has comments regarding the rejection, 
please file an Argument within 60 days of the issue date of this 
Notice. 

15 GROUNDS 

1. The inventions defined in the following claims of the present 
application could have been made easily by those having ordinary skill 
in the art, based on the inventions disclosed in the following 

20 publications, which were distributed either domestically or overseas, 
or inventions that were made available to the public via 
telecommunications lines, prior to the filing date of the present 
application. Therefore, patent rights cannot be granted, according 
to the regulations prescribed in Article 29, Paragraph 2 of the Patent 

25 Law. 

NOTES (please refer to the List of Cited References) 

Claims: 1 through 18 

30 Cited References: 1 through 3 

Improving resolution (obtaining output having greater numbers of 
pixels) by motion compensating interpolation is a well known 
technique. Weighting and synthesizing motion compensated 
35 interpolated output and intra frame interpolated output based on 



correlation values is also well known, as disclosed in Cited 
References 1 and 2. The type of motion compensating interpolation 
to be applied to improve resolution may be selected as necessary from 
among well known techniques by those skilled in the art. It would 
5 have been easy for those skilled in the art to apply the four sided 
patch disclosed in Cited Reference 3. Accordingly, no inventive step 
is recognized for inventions defined in Claims 1 through 18. 

List of Cited References 

1. Japanese Unexamined Patent Publication No. H8-130716 (refer to 
Figures 1 and 3, and the descriptions related thereto) 

2. Japanese Unexamined Patent Publication No. 2001-086508 (refer to 
Figures 1, 2, and 8, and the descriptions related thereto) 

3. Y. Nakazawa, T. Komatsu, T. Saito, "Obtainment of Highly Detailed 
Digital Images by Frame Integration", Journal of ITE, Vol. 40, 
No. 3 , pp. 299-308, The Institute of Image Information and 
Television Engineers, 1995 

20 2. The recitations of the claims of the present application do not 
satisfy the requirements prescribed in Article 36, Paragraph 6, 
Number 2 of the Patent Law in the following points. 

NOTES 

25 

(1) The recitation "acquiring a first interpolated frame whose 
resolution is higher than each of said frames, by performing 
interpolation either on the image within said second patch of 
said other frame or on the image within said second patch of 

30 said other frame and image within said reference patch of said 

reference frame, based on said correspondent relationship" is 
technically unclear regarding how the correspondent 
relationship is used to perform interpolating calculations. 
Therefore, what the obtained "first interpolated frame" is, 

35 is technically unclear. 



10 



15 



(2) The recitation "acquiring a second interpolated frame whose 
resolution is higher than each of said frames, by performing 
interpolation on the image within said reference patch of said 
reference frame" is technically unclear as to what type of 
interpolating calculation is performed. Therefore, what the 
obtained "second interpolated frame" is, is technically unclear 

(3) Because (1) and (2) above are unclear, and the relationship 
between the "first interpolated frame" and the "second 
interpolated frame" is unclear, the technical meaning of 
weighting and synthesizing these two frames based on a 
correlation value is unclear. 

(4) Claims 3, 9, and 15 are unclear as to what type of "filtering" 
is performed. The term "filtering" has many technical meanings, 
and the effects thereof differ completely depending on the 
content of the "filtering" operation. Therefore, the 
inventions defined therein cannot be technically specified. 

(5) Claims 6, 12, and 18 do not clearly define the "at least one 
component that constitutes said frame". For example, the pixel 
value of a single pixel within a frame is a "component that 
constitutes said frame". Therefore, the technical 
significance of the "at least one component that constitutes 
said frame" is unclear. (The "component" cannot be interpreted 
to refer to the limited meaning described in paragraph [0022] ) 

Accordingly, the inventions defined in Claims 1 through 18 are 
indefinite . 

Record of Prior Art Document Search Results 

• Field of Search: IPC 7 th Edition 

H04N7/00-7/01 

• Prior Art Documents 

Japanese Unexamined Patent Publication No. Hll-308577 

Japanese Unexamined Patent Publication No. 2000-354244 

Japanese Unexamined Patent Publication No. 2000-194845 



The Record of Prior Art Document Search Results does not constitute 
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[Name of Document] 
[Addressee] 
[Indication of Case] 

[Application Number] 
[Applicant] 

[ID Number] 

[Name] 
[Representative] 

[ID Number] 

[Patent Attorney] 

[Name] 
[Submission Number] 
[Amendment 1] 

[Document to be Amended] 

[Item to be Amended] 

[Manner of Amendment] 

[Contents of Amendment] 
[Scope of Demand for Patent] 
[Claim 1] 

A video image synthesis method comprising the steps of: 

sampling two contiguous frames from a video image; 

placing a reference patch comprising one or a plurality of 
rectangular areas on one of said two frames which is used as a 
reference frame, then placing on the other of said two frames a second 
patch which is the same as said reference patch, then moving and/or 
deforming said second patch in said other frame so that an image within 
said second patch coincides with an image within said reference patch, 
and estimating a correspondent relationship between a pixel within 
said second patch on said other frame and a pixel within said reference 
patch on said reference frame, based on said second patch after the 
movement and/or deformation and on said reference patch; 

acquiring a first interpolated frame whose resolution is 
higher than each of said frames, by performing interpolation either 
on the image within said second patch of said other frame 



corresponding to the reference patch or on the image within said 
second patch of said other frame corresponding to the 'reference patch 
and the image within said reference patch of said reference frame 
such that the pixel values of each pixel therein are assigned to pixel 
5 positions of an integrated image having a greater number of pixels 
than each of said frames , baocd on employing said correspondent 
relationship; 

acquiring a second interpolated frame whose resolution is 
higher than each of said frames, by performing interpolation on the 
10 image within said reference patch of said reference frame such that 
the pixel values of each pixel therein are assigned to pixel positions 
of the integrated image ; 

acquiring a coordinate-transformed frame by transforming 
coordinates of the image within said second patch of said other frame 
15 corresponding to the reference patch to a coordinate space of said 
reference frame, based on said correspondent relationship; 

computing a correlation value that represents a correlation 
between the image within the patch of said coordinate-transformed 
frame and the image within said reference patch of said reference 
20 frame; 

acquiring a weighting coefficient that makes a weight of said 
first interpolated frame greater as said correlation becomes greater, 
when synthesizing said first interpolated frame and second 
interpolated frame, based on said correlation value; and 
25 acquiring a synthesized frame by weighting and synthesizing 

the pixel values of each pixel within said first and second 
interpolated frames, based on said weighting coefficient. 
[Claim 2] 

A video image synthesis method comprising the steps of: 
30 sampling three or more contiguous frames from a video image; 

placing a reference patch comprising one or a plurality of 
rectangular areas on one of said three or more frames which is used 
as a reference frame, then respectively placing on the others of said 
three or more frames patches which are the same as said reference 
35 patch, then moving and/or deforming said patches in said other frames 



so that an image within the patch of each of said other frames 
coincides with an image within said reference patch, and respectively 
estimating correspondent relationships between pixels within the 
patches of said other frames and a pixel within said reference patch 
5 of said reference frame, based on the patches of said other frames 
after the movement and/or deformation and on said reference patch; 

acquiring a plurality of first interpolated frames whose 
resolution is higher than each of said frames, by performing 
interpolation either on the image within the patch of each of said 

10 plurality of other frames corresponding to the reference patch or 
on the image within the patch of each of said other frames 
corresponding to the reference patch and the image within said 
reference patch of said reference frame such that the pixel values 
of each pixel therein are assigned to pixel positions of an integrated 

15 image having a greater number of pixels than each of said frames , 
baocd on employing said correspondent relationships; 

acquiring one or a plurality of second interpolated frames 
whose resolution is higher than each of said frames and which are 
correlated with said plurality of first interpolated frames, by 

20 performing interpolation on the image within said reference patch 
of said reference frame such that the pixel values of each pixel 
therein are assigned to pixel positions of the integrated image ; 

acquiring a plurality of coordinate-transformed frames by 
transforming coordinates of the images within the patches of said 

25 other frames corresponding to the reference patch to a coordinate 
space of said reference frame, based on said correspondent 
relationships; 

computing correlation values that represent a correlation 
between the image within the patch of each of said 
30 coordinate-transformed frames and the image within said reference 
patch of said reference frame; 

acquiring weighting coefficients that make a weight of said 
first interpolated frame greater as said correlation becomes greater, 
when synthesizing said first interpolated frame and second 
35 interpolated frame, based on said correlation values; and 



acquiring intermediate synthesized frames by weighting and 
synthesizing the pixel values of each pixel within said first and 
second interpolated frames that correspond to each other on the basis 
of said weighting coefficients, and acquiring a synthesized frame 
5 by synthesizing said intermediate synthesized frames. 
[Claim 3] 

The synthesis method as set forth in either one of Claim 1 and 
Claim 2, wherein when said correlation value has been computed for 
each of the pixels and/or each local region that constitute each of 
10 said frames, 

said correlation value is filtered such that changes in the 
correlation values become smooth to compute a filtered correlation 
value, and said weighting coefficient is acquired based on said 
filtered correlation value. 
15 [Claim 4] 

The synthesis method as set forth in any one of Claims 1, 2, 
and 3, wherein when said correlation value has been computed for each 
of the pixels and/or each local region that constitute each of said 
frames, 

20 said weighting coefficient is interpolated such that the 

weighting coefficient for each pixel is assigned to pixel positions 
of the integrated image to acquire weighting coefficients for all 
pixels that constitute said first and second interpolated frames. 
[Claim 5] 

25 The synthesis method as set forth in any one of Claims 1 through 

4, wherein said weighting coefficient is acquired by referring to 
a nonlinear graph in which said correlation value is represented in 
the horizontal axis and said weighting coefficient in the vertical 
axis . 

30 [Claim 6] 

The synthesis method as set forth in an one of Claims 1 through 

5, wherein the estimation of said correspondent relationship, 
acquisition of said first interpolated frame, acquisition of said 
second interpolated frame, acquisition of said 

35 coordinate-transformed frame, computation of said correlation value, 



acquisition of said weighting coefficient, and acquisition of said 
synthesized frame are performed by employing at least one of a 
brightness component and a color component that constitutes said 
frame . 
5 [Claim 7] 

A video image synthesizer comprising: 

sampling means for sampling two contiguous frames from a video 

image; 

correspondent relationship estimation means for placing a 

10 reference patch comprising one or a plurality of rectangular areas 
on one of said two frames which is used as a reference frame, then 
placing on the other of said two frames a second patch which is the 
same as said reference patch, then moving and/or deforming said second 
patch in said other frame so that an image within said second patch 

15 coincides with an image within said reference patch, and estimating 
a correspondent relationship between a pixel within said second patch 
on said other frame and a pixel within said reference patch on said 
reference frame, based on said second patch after the movement and/or 
deformation and on said reference patch; 

20 first interpolation means for acquiring a first interpolated 

frame whose resolution is higher than each of said frames, by 
performing interpolation either on the image within said second patch 
of said other frame corresponding to the reference patch or on the 
image within said second patch of said other frame corresponding to 

25 the reference patch and the image within said reference patch of said 
reference frame such that the pixel values of each pixel therein are 
assigned to pixel positions of an integrated image having a greater 
number of pixels than each of said frames , based on employing said 
correspondent relationship; 

30 second interpolation means for acquiring a second interpolated 

frame whose resolution is higher than each of said frames, by 
performing interpolation on the image within said reference patch 
of said reference frame such that the pixel values of each pixel 
therein are assigned to pixel positions of the integrated image ; 

35 coordinate transformation means for acquiring a 



coordinate-transformed frame by transforming coordinates of the 
image within said second patch of said other frame corresponding to 
the reference patch to a coordinate space of said reference frame, 
based on said correspondent relationship; 
5 correlation-value computation means for computing a 

correlation value that represents a correlation between the image 
within the patch of said coordinate-transformed frame and the image 
within said reference patch of said reference frame; 

weighting-coefficient acquisition means for acquiring a 

10 weighting coefficient that makes a weight of said first interpolated 
frame greater as said correlation becomes greater, when synthesizing 
said first interpolated frame and second interpolated frame, based 
on said correlation value; and 

synthesis means for acquiring a synthesized frame by weighting 

15 and synthesizing the pixel values of each pixel of said first and 
second interpolated frames, based on said weighting coefficient. 
[Claim 8] 

A video image synthesizer comprising: 

sampling means for sampling three or more contiguous frames 

20 from a video image; 

correspondent relationship estimation means for placing a 
reference patch comprising one or a plurality of rectangular areas 
on one of said three or more frames which is used as a reference frame, 
then respectively placing on the others of said three or more frames 

25 patches which are the same as said reference patch, then moving and/or 
deforming said patches in said other frames so that an image within 
the patch of each of said other frames coincides with an image within 
said reference patch, and respectively estimating correspondent 
relationships between pixels within the patches of said other frames 

30 and a pixel within said reference patch of said reference frame, based 
on the patches of said other frames after the movement and/or 
deformation and on said reference patch; 

first interpolation means for acquiring a plurality of first 
interpolated frames whose resolution is higher than each of said 

35 frames, by performing interpolation either on the image within the 



patch of each of said plurality of other frames corresponding to the 
reference patch or on the image within the patch of each of said other 
frames corresponding to the reference patch and the image within said 
reference patch of said reference frame such that the pixel values 
5 of each pixel therein are assigned to pixel positions of an integrated 
image having a greater number of pixels than each of said frames , 
based on employing said correspondent relationships; 

second interpolation means for acquiring one or a plurality 
of second interpolated frames whose resolution is higher than each 
10 of said frames and which are correlated with said plurality of first 
interpolated frames, by performing interpolation on the image within 
said reference patch of said reference frame such that the pixel 
values of each pixel therein are assigned to pixel positions of the 
integrated image ; 

15 coordinate transformation means for acquiring a plurality of 

coordinate-transformed frames by transforming coordinates of the 
images within the patches of said other frames corresponding to the 
reference patch to a coordinate space of said reference frame, based 
on said correspondent relationships; 

20 correlation-value computation means for computing 

correlation values that represent a correlation between the image 
within the patch of each of said coordinate-transformed frames and 
the image within said reference patch of said reference frame; 
weighting-coefficient acquisition means for acquiring 

25 weighting coefficients that make a weight of said first interpolated 
frame greater as said correlation becomes greater, when synthesizing 
said first interpolated frame and second interpolated frame, based 
on said correlation values; and 

synthesis means for acquiring intermediate synthesized frames 

30 by weighting and synthesizing the pixel values of each pixel within 
said first and second interpolated frames that correspond to each 
other on the basis of said weighting coefficients, and acquiring a 
synthesized frame by synthesizing said intermediate synthesized 
frames . 

35 [Claim 9] 



The video image synthesizer as set forth in either one of Claim 
7 and Claim 8, wherein when said correlation value has been computed 
for each of pixels and/or each local region that constitute each of 
said frames, 

5 said synthesizer further comprises means for filtering said 

correlation value such that changes in the correlation values become 
smooth to compute a filtered correlation value, and said 
weighting-coefficient acquisition means acquires said weighting 
coefficient, based on said filtered correlation value. 
10 [Claim 10] 

The video image synthesizer as set forth in any one of Claim 
7, Claim 8, and Claim 9, wherein when said correlation value has been 
computed for each of the pixels and/or each local region that 
constitute each of said frames, 

15 said weighting-coefficient acquisition means performs 

interpolation on said weighting coefficient such that the weighting 
coefficient for each pixel is assigned to pixel positions of the 
integrated image , thereby acquiring weighting coefficients for all 
pixels that constitute said first and second interpolated frames., 

20 [Claim 11] 

The video image synthesizer as set forth in any one of Claims 
7 through 10, wherein said weighting-coefficient acquisition means 
acquires said weighting coefficient by referring to a nonlinear graph 
in which said correlation value is represented in the horizontal axis 
25 and said weighting coefficient in the vertical axis. 
[Claim 12] 

The video image synthesizer as set forth in any one of Claims 
7 through 11, wherein said correspondent relationship estimation 
means, said first interpolation means, said second interpolation 

30 means, said coordinate transformation means, said correlation-value 
computation means, said weighting-coefficient acquisition means, 
and said synthesis means perform the estimation of said correspondent 
relationship, acquisition of said first interpolated frame, 
acquisition of said second interpolated frame, acquisition of said 

35 coordinate-transformed frame, computation of said correlation value, 



acquisition of said weighting coefficient, and acquisition of said 
synthesized frame, by employing at least one of a brightness component 
and a color component that constitutes said frame. 
[Claim 13] 

5 A program for causing a computer to execute a video image 

synthesis method comprising: 

a procedure of sampling two contiguous frames from a video 

image; 

a procedure of placing a reference patch comprising one or a 
10 plurality of rectangular areas on one of said two frames which is 
used as a reference frame, then placing on the other of said two frames 
a second patch which is the same as said reference patch, then moving 
and/or deforming said second patch in said other frame so that an 
image within said second patch coincides with an image within said 
15 reference patch, and estimating a correspondent relationship between 
a pixel within said second patch on said other frame and a pixel within 
said reference patch on said reference frame, based on said second 
patch after the movement and/or deformation and on said reference 
patch; 

20 a procedure of acquiring a first interpolated frame whose 

resolution is higher than each of said frames, by performing 
interpolation either on the image within said second patch of said 
other frame corresponding to the reference patch or on the image 
within said second patch of said other frame corresponding to the 

25 reference patch and the image within said reference patch of said 
reference frame such that the pixel values of each pixel therein are 
assigned to pixel positions of an integrated image having a greater 
number of pixels than each of said frames , based on employing said 
correspondent relationship; 

30 a procedure of acquiring a second interpolated frame whose 

resolution is higher than each of said frames, by performing 
interpolation on the image within said reference patch of said 
reference frame such that the pixel values of each pixel therein are 
assigned to pixel positions of the integrated image ; 

35 a procedure of acquiring a coordinate-transformed frame by 



transforming coordinates of the image within said second patch of 
said other frame corresponding to the reference patch to a coordinate 
space of said reference frame, based on said correspondent 
relationship; 

5 a procedure of computing a correlation value that represents 

a correlation between the image within the patch of said 
coordinate-transformed frame and the image within said reference 
patch of said reference frame; 

a procedure of acquiring a weighting coefficient that makes 
10 a weight of said first interpolated frame greater as said correlation 
becomes greater, when synthesizing said first interpolated frame and 
second interpolated frame, based on said correlation value; and 

a procedure of acquiring a synthesized frame by weighting and 
synthesizing the pixel values of each pixel within said first and 
15 second interpolated frames, based on said weighting coefficient. 
[Claim 14] 

A program for causing a computer to execute a video image 
synthesis method comprising: 

a procedure of sampling three or more contiguous frames from 
20 a video image; 

a procedure of placing a reference patch comprising one or a 
plurality of rectangular areas on one of said three or more frames 
which is used as a reference frame, then respectively placing on the 
others of said three or more frames patches which are the same as 
25 said reference patch, then moving and/or deforming said patches in 
said other frames so that an image within the patch of each of said 
other frames coincides with an image within said reference patch, 
and respectively estimating correspondent relationships between 
pixels within the patches of said other frames and a pixel within 
30 said reference patch of said reference frame, based on the patches 
of said other frames after the movement and/or deformation and on 
said reference patch; 

a procedure of acquiring a plurality of first interpolated 
frames whose resolution is higher than each of said frames, by 
35 performing interpolation either on the image within the patch of each 



of said plurality of other frames corresponding to the reference patch 
or on the image within the patch of each of said other frames 
corresponding to the reference patch and the image within said 
reference patch of said reference frame such that the pixel values 
5 of each pixel therein are assigned to pixel positions of an integrated 
image having a greater number of pixels than each of said frames , 
based on employing said correspondent relationships; 

a procedure of acquiring one or a plurality of second 
interpolated frames whose resolution is higher than each of said 
10 frames and which are correlated with said plurality of first 

interpolated frames, by performing interpolation on the image within 
said reference patch of said reference frame such that the pixel 
values of each pixel therein are assigned to pixel positions of the 
integrated image ; 

15 a procedure of acquiring a plurality of coordinate-transformed 

frames by transforming coordinates of the images within the patches 
of said other frames corresponding to the reference patch to a 
coordinate space of said reference frame, based on said correspondent 
relationships; 

20 a procedure of computing correlation values that represent a 

correlation between the image within the patch of each of said 
coordinate-transformed frames and the image within said reference 
patch of said reference frame; 

a procedure of acquiring weighting coefficients that make a 

25 weight of said first interpolated frame greater as said correlation 
becomes greater, when synthesizing said first interpolated frame and 
second interpolated frame, based on said correlation values; and 
a procedure of acquiring intermediate synthesized frames by 
weighting and synthesizing the pixel values of each pixel within said 

30 first and second interpolated frames that correspond to each other 
on the basis of said weighting coefficients, and acquiring a 
synthesized frame by synthesizing said intermediate synthesized 
frames. 
[Claim 15] 

35 The program as set forth in either one of Claim 13 and Claim 



14, wherein when said correlation value has been computed for each 
pixel and/or each local region that constitutes each of said frames, 
said method further comprises a procedure of filtering said 
correlation value such that changes in the correlation values become 
5 smooth to compute a filtered correlation value, and said 

weighting-coefficient acquisition procedure is a procedure of 
acquiring said weighting coefficient, based on said filtered 
correlation value. 
[Claim 16] 

10 The program as set forth in any one of Claim 13, Claim 14, and 

Claim 15, wherein when said correlation value has been computed for 
each of the pixels and/or each local region that constitute each of 
said frames, 

said weighting-coefficient acquisition procedure is a 
15 procedure of performing interpolation on said weighting coefficient 
such that the weighting coefficient for each pixel is assigned to 
pixel positions of the integrated image and acquiring weighting 
coefficients for all pixels that constitute said first and second 
interpolated frames. 
20 [Claim 17] 

The program as set forth in any one of Claims 13 through 16, 
wherein said weighting-coefficient acquisition procedure is a 
procedure of acquiring said weighting coefficient by referring to 
a nonlinear graph in which said correlation value is represented in 
25 the horizontal axis and said weighting coefficient in the vertical 
axis . 
[Claim 18] 

The program as set forth in any one of Claims 13 through 17, 
wherein said correspondent relationship estimation procedure, said 

30 first interpolated frame acquisition procedure, said second 
interpolated frame acquisition procedure, said 
coordinate-transformed frame acquisition procedure, said 
correlation value computation procedure, said weighting-coefficient 
acquisition procedure, and said synthesized-f rame acquisition 

35 procedure are procedures of performing the estimation of said 



correspondent relationship, acquisition of said first interpolated 
frame, acquisition of said second interpolated frame, acquisition 
of said coordinate-transformed frame, computation of said 
correlation value, acquisition of said weighting coefficient, and 
5 acquisition of said synthesized frame by employing at least one of 
a brightness component and a color component that constitutes said 
frame, respectively. 
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15 Note that the contents of the Argument and the Amendment were 

considered, but no basis sufficient to overturn the Grounds for 
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Remarks 

20 

When interpolating video having large movements or video having small 
movements therein, it is a well known technique to weight an 
interpolating method better suited to the video with a larger weight, 
as disclosed in Cited References 1 and 2. In motion compensating 

25 interpolation based on amounts of change accompanying parallel motion, 
it is not possible to obtain motion compensated interpolated signals 
that enable obtainment of high quality images in cases that subjects 
are enlarged, reduced, or rotated between two fields. Cited 
Reference 3 discloses the use of a four sided patch method for such 

3 0 cases (refer to page 3 01, "Estimation of Motion using Four Sided 
Patches") . Accordingly, it would have been easy for those skilled 
in the art to apply the four sided patch method disclosed in Cited 
Reference 3 to the inventions disclosed in Cited References 1 and 
2, to achieve the inventions defined in Claims 1, 2, 7, 8, 13, and 

35 14 of the present application. In addition, no novel technical 



features are recognized in the inventions of the dependent claims. 

In the Argument, the Applicant argued that "The invention disclosed 
in Cited Reference 3 deals with video images having small movements. 
5 On the other hand, the inventions disclosed in Cited References 1 
and 2 deal with video images having large movements. It would not 
have been easy for those skilled in the art to conceive of applying 
the invention disclosed in Cited Reference 3 , which calculates motion 
compensating interpolation signals (that is, the first interpolated 

10 frame) for video images having small movements, o the inventions 
disclosed in Cited References 1 and 2, which deal with video images 
having large movements" . However, in the inventions disclosed in 
Cited Reference 1 and 2 , the weighting coefficient k changes according 
to the degree of movement. Therefore, the argument that "the 

15 inventions disclosed in Cited References 1 and 2 deal with video 
images having large movements" cannot be accepted. In addition, page 
3 05 of Cited Reference 3 mentions cases in which the degree of movement 
is large . Therefore, the argument that "invention disclosed in Cited 
Reference 3 deals with video images having small movements" has no 

20 basis, and cannot be accepted. 

As described above, no inventive step is recognized for the inventions 
defined in Claims 1 through 18 of the present application. 

25 

If the applicant is dissatisfied with this decision, an appeal may 
be filed with the Commissioner of the Patent Office within 3 0 days 
(90 days for foreign residents) of delivery of a copy of this decision 
(Article 121, Paragraph 1 of the Patent Law) . 

30 

(Teaching in accordance with Article 46, Paragraph 2 of the 
Administrative Case Litigation Law) 

It is not possible to file a suit to revoke this decision without 
filing the aforementioned appeal. In the case that the results of 
35 the aforementioned appeal are not satisfactory, then a suit may be 



filed to revoke the appeal decision (Article 178, Paragraph 6 of the 
Patent Law) . 
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[Claim 1] 

A video image synthesis method comprising the steps of: 

sampling two contiguous frames from a video image; 

placing a reference patch comprising one or a plurality of 
rectangular areas on one of said two frames which is used as a 
reference frame, then placing on the other of said two frames a second 
patch which is the same as said reference patch, then moving and/or 
deforming said second patch in said other frame so that an image within 
said second patch coincides with an image within said reference patch, 
and estimating a correspondent relationship between a pixel within 
said second patch on said other frame and a pixel within said reference 
patch on said reference frame, based on said second patch after the 
movement and/or deformation and on said reference patch; 

acquiring a first interpolated frame whose resolution is 
higher than each of said frames, by performing interpolation either 
on the image within said second patch of said other frame 



corresponding to the reference patch or on the image within said 
second patch of said other frame corresponding to the reference patch 
and the image within said reference patch of said reference frame 
such that the pixel values of each pixel therein are assigned to real 
5 number coordinate pixel positions of an integrated image having a 
greater number of pixels than each of said frames, employing said 
correspondent relationship; 

acquiring a second interpolated frame whose resolution is 
higher than each of said frames, by performing interpolation on the 

10 image within said reference patch of said reference frame such that 
the pixel values of each pixel therein are assigned to the real number 
coordinate pixel positions of the integrated image; 

acquiring a coordinate-transformed frame by transforming 
coordinates of the image within said second patch of said other frame 

15 corresponding to the reference patch to a coordinate space of said 
reference frame, based on said correspondent relationship; 

computing a correlation value that represents a correlation 
between the image within the patch of said coordinate-transformed 
frame and the image within said reference patch of said reference 

20 frame; 

acquiring a weighting coefficient that makes a weight of said 
first interpolated frame greater as said correlation becomes greater, 
when synthesizing said first interpolated frame and second 
interpolated frame, based on said correlation value; and 

25 acquiring a synthesized frame by weighting and synthesizing 

the pixel values of each pixel within said first and second 
interpolated frames, based on said weighting coefficient , and 
converting the weighted and synthesized pixel values to an integer 
coordinate system of the integrated image . 

30 [Claim 2] 

A video image synthesis method comprising the steps of: 
sampling three or more contiguous frames from a video image; 
placing a reference patch comprising one or a plurality of 
rectangular areas on one of said three or more frames which is used 
35 as a reference frame, then respectively placing on the others of said 



three or more frames patches which are the same as said reference 
patch, then moving and/or deforming said patches in said other frames 
so that an image within the patch of each of said other frames 
coincides with an image within said reference patch, and respectively 
5 estimating correspondent relationships between pixels within the 
patches of said other frames and a pixel within said reference patch 
of said reference frame, based on the patches of said other frames 
after the movement and/or deformation and on said reference patch; 
acquiring a plurality of first interpolated frames whose 

10 resolution is higher than each of said frames, by performing 

interpolation either on the image within the patch of each of said 
plurality of other frames corresponding to the reference patch or 
.on the image within the patch of each of said other frames 
corresponding to the reference patch and the image within said 

15 reference patch of said reference frame such that the pixel values 
of each pixel therein are assigned to real number coordinate pixel 
positions of an integrated image having a greater number of pixels 
than each of said frames, employing said correspondent relationships; 

acquiring one or a plurality of second interpolated frames 

20 whose resolution is higher than each of said frames and which are 
correlated with said plurality of first interpolated frames, by 
performing interpolation on the image within said reference patch 
of said reference frame such that the pixel values of each pixel 
therein are assigned to the real number coordinate pixel positions 

25 of the integrated image; 

acquiring a plurality of coordinate-transformed frames by 
transforming coordinates of the images within the patches of said 
other frames corresponding to the reference patch to a coordinate 
space of said reference frame, based on said correspondent 

30 relationships; 

computing correlation values that represent a correlation 
between the image within the patch of each of said 
coordinate-transformed frames and the image within said reference 
patch of said reference frame; 

35 acquiring weighting coefficients that make a weight of said 



first interpolated frame greater as said correlation becomes greater, 
when synthesizing said first interpolated frame and second 
interpolated frame, based on said correlation values; and 

acquiring intermediate synthesized frames by weighting and 
5 synthesizing the pixel values of each pixel within said first and 
second interpolated frames that correspond to each other on the basis 
of said weighting coefficients , converting the weighted and 
synthesized pixel values to an integer coordinate system of the 
integrated image , and acquiring a synthesized frame by synthesizing 
10 said intermediate synthesized frames. 
[Claim 3] 

The synthesis method as set forth in either one of Claim 1 and 
Claim 2, wherein when said correlation value has been computed for 
each of the pixels and/or each local region that constitute each of 
15 said frames, 

said correlation value is filtered such that changes in the 
correlation values become smooth to compute a filtered correlation 
value, and said weighting coefficient is acquired based on said 
filtered correlation value. 
20 [Claim 4] 

The synthesis method as set forth in any one of Claims 1, 2, 
and 3, wherein when said correlation value has been computed for each 
of the pixels and/or each local region that constitute each of said 
frames, 

25 said weighting coefficient is interpolated such that the pixel 

values of each pixel therein are assigned to the real number 
coordinate pixel positions of the integrated image to acquire 
weighting coefficients for all pixels that constitute said first and 
second interpolated frames. 

30 [Claim 5] 

The synthesis method as set forth in any one of Claims 1 through 
4, wherein said weighting coefficient is acquired by referring to 
a nonlinear graph in which said correlation value is represented in 
the horizontal axis and said weighting coefficient in the vertical 
35 axis. 



[Claim 6] 

The synthesis method as set forth in an one of Claims 1 through 
5, wherein the estimation of said correspondent relationship, 
acquisition of said first interpolated frame, acquisition of said 
5 second interpolated frame, acquisition of said 

coordinate-transformed frame, computation of said correlation value, 
acquisition of said weighting coefficient, and acquisition of said 
synthesized frame are performed by employing at least one of a 
brightness component and a color component that constitutes said 
10 frame. 

[Claim 7] 

A video image synthesizer comprising: 

sampling means for sampling two contiguous frames from a video 

image; 

15 correspondent relationship estimation means for placing a 

reference patch comprising one or a plurality of rectangular areas 
on one of said two frames which is used as a reference frame, then 
placing on the other of said two frames a second patch which is the 
same as said reference patch, then moving and/or deforming said second 

20 patch in said other frame so that an image within said second patch 
coincides with an image within said reference patch, and estimating 
a correspondent relationship between a pixel within said second patch 
on said other frame and a pixel within said reference patch on said 
reference frame, based on said second patch after the movement and/or 

25 deformation and on said reference patch; 

first interpolation means for acquiring a first interpolated 
frame whose resolution is higher than each of said frames, by 
performing interpolation either on the image within said second patch 
of said other frame corresponding to the reference patch or on the 

30 image within said second patch of said other frame corresponding to 
the reference patch and the image within said reference patch of said 
reference frame such that the pixel values of each pixel therein are 
assigned to real number coordinate pixel positions of an integrated 
image having a greater number of pixels than each of said frames, 

35 employing said correspondent relationship; 



second interpolation means for acquiring a second interpolated 
frame whose resolution is higher than each of said frames, by 
performing interpolation on the image within said reference patch 
of said reference frame such that the pixel values of each pixel 
5 therein are assigned to the real number coordinate pixel positions 
of the integrated image; 

coordinate transformation means for acquiring a 
coordinate-transformed frame by transforming coordinates of the 
image within said second patch of said other frame corresponding to 
10 the reference patch to a coordinate space of said reference frame, 
based on said correspondent relationship; 

correlation-value computation means for computing a 
correlation value that represents a correlation between the image 
within the patch of said coordinate-transformed frame and the image 
15 within said reference patch of said reference frame; 

weighting-coefficient acquisition means for acquiring a 
weighting coefficient that makes a weight of said first interpolated 
frame greater as said correlation becomes greater, when synthesizing 
said first interpolated frame and second interpolated frame, based 
20 on said correlation value; and 

synthesis means for acquiring a synthesized frame by weighting 
and synthesizing the pixel values of each pixel of said first and 
second interpolated frames, based on said weighting coefficient , and 
converting the weighted and synthesized pixel values to an integer 
25 coordinate system of the integrated image . 
[Claim 8] 

A video image synthesizer comprising: 

sampling means for sampling three or more contiguous frames 
from a video image; 

30 correspondent relationship estimation means for placing a 

reference patch comprising one or a plurality of rectangular areas 
on one of said three or more frames which is used as a reference frame, 
then respectively placing on the others of said three or more frames 
patches which are the same as said reference patch, then moving and/or 

35 deforming said patches in said other frames so that an image within 



the patch of each of said other frames coincides with an image within 
said reference patch, and respectively estimating correspondent 
relationships between pixels within the patches of said other frames 
and a pixel within said reference patch of said reference frame, based 
5 on the patches of said other frames after the movement and/or 
deformation and on said reference patch; 

first interpolation means for acquiring a plurality of first 
interpolated frames whose resolution is higher than each of said 
frames, by performing interpolation either on the image within the 

10 patch of each of said plurality of other frames corresponding to the 
reference patch or on the image within the patch of each of said other 
frames corresponding to the reference patch and the image within said 
reference patch of said reference frame such that the pixel values 
of each pixel therein are assigned to real number coordinate pixel 

15 positions of an integrated image having a greater number of pixels 
than each of said frames, employing said correspondent relationships; 

second interpolation means for acquiring one or a plurality 
of second interpolated frames whose resolution is higher than each 
of said frames and which are correlated with said plurality of first 

20 interpolated frames, by performing interpolation on the image within 
said reference patch of said reference frame such that the pixel 
values of each pixel therein are assigned to the real number 
coordinate pixel positions of the integrated image; 

coordinate transformation means for acquiring a plurality of 

25 coordinate-transformed frames by transforming coordinates of the 
images within the patches of said other frames corresponding to the 
reference patch to a coordinate space of said reference frame, based 
on said correspondent relationships; 

correlation-value computation means for computing 

30 correlation values that represent a correlation between the image 
within the patch of each of said coordinate-transformed frames and 
the image within said reference patch of said reference frame; 

weighting-coefficient acquisition means for acquiring 
weighting coefficients that make a weight of said first interpolated 

35 frame greater as said correlation becomes greater, when synthesizing 



said first interpolated frame and second interpolated frame, based 
on said correlation values; and 

synthesis means for acquiring intermediate synthesized frames 
by weighting and synthesizing the pixel values of each pixel within 
5 said first and second interpolated frames that correspond to each 
other on the basis of said weighting coefficients, converting the 
weighted and synthesized pixel values to an integer coordinate system 
of the integrated image, and acquiring a synthesized frame by 
synthesizing said intermediate synthesized frames. 
10 [Claim 9] 

The video image synthesizer as set forth in either one of Claim 
7 and Claim 8, wherein when said correlation value has been computed 
for each of pixels and/or each local region that constitute each of 
said frames, 

15 said synthesizer further comprises means for filtering said 

correlation value such that changes in the correlation values become 
smooth to compute a filtered correlation value, and said 
weighting-coefficient acquisition means acquires said weighting 
coefficient, based on said filtered correlation value. 

20 [Claim 10] 

The video image synthesizer as set forth in any one of Claim 
7, Claim 8, and Claim 9, wherein when said correlation value has been 
computed for each of the pixels and/or each local region that 
constitute each of said frames, 

25 said weighting-coefficient acquisition means performs 

interpolation on said weighting coefficient such that the pixel 
values of each pixel therein are assigned to the real number 
coordinate pixel positions of the integrated image, thereby acquiring 
weighting coefficients for all pixels that constitute said first and 

30 second interpolated frames. 
[Claim 11] 

The video image synthesizer as set forth in any one of Claims 
7 through 10, wherein said weighting-coefficient acquisition means 
acquires said weighting coefficient by referring to a nonlinear graph 
35 in which said correlation value is represented in the horizontal axis 



and said weighting coefficient in the vertical axis. 
[Claim 12] 

The video image synthesizer as set forth in any one of Claims 
7 through 11, wherein said correspondent relationship estimation 
5 means, said first interpolation means, said second interpolation 
means, said coordinate transformation means, said correlation-value 
computation means, said weighting-coefficient acquisition means, 
and said synthesis means perform the estimation of said correspondent 
relationship, acquisition of said first interpolated frame, 
10 acquisition of said second interpolated frame, acquisition of said 
coordinate-transformed frame, computation of said correlation value, 
acquisition of said weighting coefficient, and acquisition of said 
synthesized frame, by employing at least one of a brightness component 
and a color component that constitutes said frame. 
15 [Claim 13] 

A program for causing a computer to execute a video image 
synthesis method comprising: 

a procedure of sampling two contiguous frames from a video 

image; 

20 a procedure of placing a reference patch comprising one or a 

plurality of rectangular areas on one of said two frames which is 
used as a reference frame, then placing on the other of said two frames 
a second patch which is the same as said reference patch, then moving 
and/or deforming said second patch in said other frame so that an 

25 image within said second patch coincides with an image within said 
reference patch, and estimating a correspondent relationship between 
a pixel within said second patch on said other frame and a pixel within 
said reference patch on said reference frame, based on said second 
patch after the movement and/or deformation and on said reference 

30 patch; 

a procedure of acquiring a first interpolated frame whose 
resolution is higher than each of said frames, by performing 
interpolation either on the image within said second patch of said 
other frame corresponding to the reference patch or on the image 
35 within said second patch of said other frame corresponding to the 



reference patch and the image within said reference patch of said 
reference frame such that the pixel values of each pixel therein are 
assigned to real number coordinate p ixel positions of an integrated 
image having a greater number of pixels than each of said frames, 
5 employing said correspondent relationship; 

a procedure of acquiring a second interpolated frame whose 
resolution is higher than each of said frames, by performing 
interpolation on the image within said reference patch of said 
reference frame such that the pixel values of each pixel therein are 

10 assigned to the real number coordinate pixel positions of the 
integrated image; 

a procedure of acquiring a coordinate-transformed frame by 
transforming coordinates of the image within said second patch of 
said other frame corresponding to the reference patch to a coordinate 

15 space of said reference frame, based on said correspondent 
relationship; 

a procedure of computing a correlation value that represents 
a correlation between the image within the patch of said 
coordinate-transformed frame and the image within said reference 
20 patch of said reference frame; 

a procedure of acquiring a weighting coefficient that makes 
a weight of said first interpolated frame greater as said correlation 
becomes greater, when synthesizing said first interpolated frame and 
second interpolated frame, based on said correlation value; and 
25 a procedure of acquiring a synthesized frame by weighting and 

synthesizing the pixel values of each pixel within said first and 
second interpolated frames, based on said weighting coefficient , and 
converting the weighted and synthesized pixel values to an integer 
coordinate system of the integrated image . 
30 [Claim 14] 

A program for causing a computer to execute a video image 
synthesis method comprising: 

a procedure of sampling three or more contiguous frames from 
a video image; 

35 a procedure of placing a reference patch comprising one or a 



plurality of rectangular areas on one of said three or more frames 
which is used as a reference frame, then respectively placing on the 
others of said three or more frames patches which are the same as 
said reference patch, then moving and/or deforming said patches in 
5 said other frames so that an image within the patch of each of said 
other frames coincides with an image within said reference patch, 
and respectively estimating correspondent relationships between 
pixels within the patches of said other frames and a pixel within 
said reference patch of said reference frame, based on the patches 

10 of said other frames after the movement and/or deformation and on 
said reference patch; 

a procedure of acquiring a plurality of first interpolated 
frames whose resolution is higher than each of said frames, by 
performing interpolation either on the image within the patch of each 

15 of said plurality of other frames corresponding to the reference patch 
or on the image within the patch of each of said other frames 
corresponding to the reference patch and the image within said 
reference patch of said reference frame such that the pixel values 
of each pixel therein are assigned to real number coordinate pixel 

20 positions of an integrated image having a greater number of pixels 
than each of said frames, employing said correspondent relationships; 

a procedure of acquiring one or a plurality of second 
interpolated frames whose resolution is higher than each of said 
frames and which are correlated with said plurality of first 

25 interpolated frames, by performing interpolation on the image within 
said reference patch of said reference frame such that the pixel 
values of each pixel therein are assigned to the real number 
coordinate pixel positions of the integrated image; 

a procedure of acquiring a plurality of coordinate-transformed 

30 frames by transforming coordinates of the images within the patches 
of said other frames corresponding to the reference patch to a 
coordinate space of said reference frame, based on said correspondent 
relationships; 

a procedure of computing correlation values that represent a 
35 correlation between the image within the patch of each of said 



coordinate-transformed frames and the image within said reference 
patch of said reference frame; 

a procedure of acquiring weighting coefficients that make a 
weight of said first interpolated frame greater as said correlation 
5 becomes greater, when synthesizing said first interpolated frame and 
second interpolated frame, based on said correlation values; and 

a procedure of acquiring intermediate synthesized frames by 
weighting and synthesizing the pixel values of each pixel within said 
first and second interpolated frames that correspond to each other 
10 on the basis of said weighting coefficients , converting the weighted 
and synthesized pixel values to an integer coordinate system of the 
integrated image , and acquiring a synthesized frame by synthesizing 
said intermediate synthesized frames. 
[Claim 15] 

15 The program as set forth in either one of Claim 13 and Claim 

14, wherein when said correlation value has been computed for each 
pixel and/or each local region that constitutes each of said frames, 
said method further comprises a procedure of filtering said 
correlation value such that changes in the correlation values become 

20 smooth to compute a filtered correlation value, and said 

weighting-coefficient acquisition procedure is a procedure of 
acquiring said weighting coefficient, based on said filtered 
correlation value. 
[Claim 16] 

25 The program as set forth in any one of Claim 13, Claim 14, and 

Claim 15, wherein when said correlation value has been computed for 
each of the pixels and/or each local region that constitute each of 
said frames, 

said weighting-coefficient acquisition procedure is a 
30 procedure of performing interpolation on said weighting coefficient 
such that the pixel values of each pixel therein are assigned to the 
real number coordinate pixel positions of the integrated image and 
acquiring weighting coefficients for all pixels that constitute said 
first and second interpolated frames. 
35 [Claim 17] 



The program as set forth in any one of Claims 13 through 16, 
wherein said weighting-coefficient acquisition procedure is a 
procedure of acquiring said weighting coefficient by referring to 
a nonlinear graph in which said correlation value is represented in 
5 the horizontal axis and said weighting coefficient in the vertical 
axis . 
[Claim 18] 

The program as set forth in any one of Claims 13 through 17, 
wherein said correspondent relationship estimation procedure, said 

10 first interpolated frame acquisition procedure, said second 
interpolated frame acquisition procedure, said 
coordinate-transformed frame acquisition procedure, said 
correlation value computation procedure, said weighting-coefficient 
acquisition procedure,, and said synthesized-f rame acquisition 

15 procedure are procedures of performing the estimation of said 

correspondent relationship, acquisition of said first interpolated 
frame, acquisition of said second interpolated frame, acquisition 
of said coordinate-transformed frame, computation of said 
correlation value, acquisition of said weighting coefficient, and 

20 acquisition of said synthesized frame by employing at least one of 
a brightness component and a color component that constitutes said 
frame, respectively. 
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10 The present application is rejected based on the following Grounds. 
In the case that the applicant has comments regarding the rejection, 
please file an Argument within 60 days of the issue date of this 
Notice . 



1. The Detailed Description of the Invention of the present 
application does not satisfy the requirements prescribed in Article 
36, Paragraph 4 of the Patent Law in the following point. 



Reviewing paragraph [0027], edge intensity greatly differs according 
to the contents of images. It is technically unclear why it can be 
said that "If the motion of a subject in each frame is small, the 
edge intensity of the reference frame and/or the other frame becomes 

25 great, but if the motion is great or complicated, it moves the contour 
of the subject and makes the edge intensity small". The causal 
relationship between the motion of a subject and edge intensity is 
unclear. In addition, even if the degree of movement is large, 
appropriate motion compensated interpolation output can be obtained 

30 if accurate motion detection is performed. Therefore, the 
relationship between the degree of movement' and the weighting 
coefficient is also unclear. Accordingly, the technical meaning of 
performing weighting and synthesis based on edge intensities is also 
unclear. 



15 



GROUNDS 



20 



35 



Accordingly, the Detailed Description of the Invention of the present 
application does not describe the inventions defined in Claims 1 
through 12 according to the regulations prescribed by the Ministry 
of Business and Industry. 

2. The recitations of the claims of the present application do not 
satisfy the requirements prescribed in Article 36, Paragraph 6, 
Number 2 of the Patent Law in the following points. 



The recitation "acquiring a first interpolated frame whose 
resolution is higher than each of said frames, by performing 
interpolation either on the image within said second patch of 
said other frame or on the image within said second patch of 
said other frame and image within said reference patch of said 
reference frame, based on said correspondent relationship" is 
technically unclear regarding how the correspondent 
relationship is used to perform interpolating calculations . 
Therefore, what the obtained "first interpolated frame" is, 
is technically unclear. 

The recitation "acquiring a second interpolated frame whose 
resolution is higher than each of said frames, by performing 
interpolation on the image within said reference patch of said 
reference frame" is technically unclear as to what type of 
interpolating calculation is performed. Therefore, what the 
obtained "second interpolated frame" is, is technically unclear 
Because (1) and (2) above are unclear, and the relationship 
between the "first interpolated frame" and the "second 
interpolated frame" is unclear, the technical meaning of 
weighting and synthesizing these two frames based on a 
correlation value is unclear. 

Claims 4, 8, and 12 do not clearly define the "at least one 
component that constitutes said frame" . For example, the pixel 
value of a single pixel within a frame is a "component that 



constitutes said frame". Therefore, the technical 

significance of the "at least one component that constitutes 
said frame" is unclear. (The "component" cannot be interpreted 
to refer to the limited meaning described in paragraph [0017] ) 

5 

Accordingly, the inventions defined in Claims 1 through 12 are 
indefinite . 

3. The inventions defined in the following claims of the present 
10 application could have been made easily by those having ordinary skill 
in the art, based on the inventions disclosed in the following 
publications, which were distributed either domestically or overseas, 
or inventions that were made available to the public via 
telecommunications lines, prior to the filing date of the present 
15 application. Therefore, patent rights cannot be granted, according 
to the regulations prescribed in Article 29, Paragraph 2 of the Patent 
Law. 

NOTES (please refer to the List of Cited References) 

20 

Claims: 1 through 12 

Cited References: 1 and 2 

Improving resolution (obtaining output having greater numbers of 
25 pixels) by movement compensating interpolation is a well known 
technique. Weighting and synthesizing movement compensated 
interpolated output and intra frame interpolated output based on 
correlation values is also well known, as disclosed in Cited Reference 
1. The type of movement compensating interpolation to be applied 
30 to improve resolution may be selected as necessary from among well 
known techniques by those skilled in the art. It would have been 
easy for those skilled in the art to apply the four sided patch 
disclosed in Cited Reference 2. Accordingly, no inventive step is 
recognized for inventions defined in Claims .1 through 12. 

35 
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[Claim 1] 

A video image synthesis method comprising the steps of: 

sampling two contiguous frames from a video image; 

placing a reference patch comprising one or a plurality of 
rectangular areas on one of said two frames which is used as a 
reference frame, then placing on the other of said two frames a second 
patch which is the same as said reference patch, then moving and/or 
deforming said second patch in said other frame so that an image within 
said second patch coincides with an image within said reference patch, 
and estimating a correspondent relationship between a pixel within 
said second patch on said other frame and a pixel within said reference 
patch on said reference frame, based on said second patch after the 
movement and/or deformation and on said reference patch; 

acquiring a first interpolated frame whose resolution is 
higher than each of said frames, by performing interpolation either 
on the image within said second patch of said other frame 



corresponding to said reference patch or on the image within said 
second patch of said other frame corresponding to said reference patch 
and the image within said reference patch of said reference frame 
such that the pixel values of each pixel therein are assigned to pixel 
5 positions of an integrated image having a greater number of pixels 
than each of said frames , baocd on employing said correspondent 
relationship; 

acquiring a second interpolated frame whose resolution is 
higher than each of said frames, by performing interpolation on the 
10 image within said reference patch of said reference frame such that 
the pixel values of each pixel therein are assigned to pixel positions 
of the integrated image ; 

acquiring edge information that represents an edge intensity 
of each pixel within the image within said reference patch of said 
15 reference frame and/or image within the patch of said other frame; 

acquiring a weighting coefficient that makes a weight of said 
first interpolated frame greater as said edge information becomes 
greater, when synthesizing said first interpolated frame and second 
interpolated frame, based on said edge information; and 
20 acquiring a synthesized frame by weighting and synthesizing 

the pixel values of each pixel within said first and second 
interpolated frames, based on said weighting coefficient. 
[Claim 2] 

A video image synthesis method comprising the steps of: 
25 sampling three or more contiguous frames from a video image; 

placing a reference patch comprising one or a plurality of 
rectangular areas on one of said three or more frames which is used 
as a reference frame, then respectively placing on the others of said 
three or more frames patches which are the same as said reference 
3 0 patch, then moving and/or deforming said patches in said other frames 
so that an image within the patch of each of said other frames 
coincides with an image within said reference patch, and respectively 
estimating correspondent relationships between pixels within the 
patches of said other frames and a pixel within said reference patch 
35 of said reference frame, based on the patches of said other frames 



after the movement and/or deformation and on said reference patch; 

acquiring a plurality of first interpolated frames whose 
resolution is higher than each of said frames, by performing 
interpolation either on the image within the patch of each of said 
5 other frames corresponding to said reference patch or on the image 
within the patch of each of said other frames corresponding to said 
reference patch and the image within said reference patch of said 
reference frame, such that the pixel values of each pixel therein 
are assigned to pixel positions of an integrated image having a 
10 greater number of pixels than each of said frames , baaed on employing 
said correspondent relationships; 

acquiring one or a plurality of second interpolated frames 
whose resolution is higher than each of said frames and which are 
correlated with said plurality of first interpolated frames, by 
15 performing interpolation on the image within said reference patch 
of said reference frame such that the pixel values of each pixel 
therein are assigned to pixel positions of the integrated image ; 

acquiring edge information that represents an edge intensity 
of each pixel within the image within said reference patch of said 

2 0 reference frame and/or image within the patch of each of said other 

frames; 

acquiring weighting coefficients that make a weight of said 
first interpolated frame greater as said edge information becomes 
greater, when synthesizing said first interpolated frame and second 
25 interpolated frame, based on said edge information; and 

acquiring intermediate synthesized frames by weighting and 
synthesizing the pixel values of each pixel within said first and 
second interpolated frames that correspond to each other on the basis 
of said weighting coefficients, and acquiring a synthesized frame 

3 0 by synthesizing said intermediate synthesized frames. 

[Claim 3] 

The synthesis method as set forth in either one of Claim 1 and 
Claim 2, wherein when said edge information has been computed for 
each of the pixels that constitute each of said frames, 
35 said weighting coefficient is interpolated such that the 



weighting coefficient for each pixel is assigned to pixel positions 
of the integrated image to acquire weighting coefficients for all 
pixels that constitute said first and second interpolated frames. 
[Claim 4] 

5 The synthesis method as set forth in any one of Claim 1, Claim 

2, and Claim 3, wherein the estimation of said correspondent 
relationship, acquisition of said first interpolated frame, 
acquisition of said second interpolated frame, acquisition of said 
edge information, acquisition of said weighting coefficient, and 
10 acquisition of said synthesized frame are performed by employing at 
least one of a brightness component and a color component that 
constitutes said frame. 
[Claim 5] 

A video image synthesizer comprising: 
15 sampling means for sampling two contiguous frames from a video 

image ; 

correspondent relationship estimation means for placing a 
reference patch comprising one or a plurality of rectangular areas 
on one of said two frames which is used as a reference frame, then 

20 placing on the other of said two frames a second patch which is the 
same as said reference patch, then moving and/or deforming said second 
patch in said other frame so that an image within said second patch 
coincides with an image within said reference patch, and estimating 
a correspondent relationship between a pixel within said second patch 

25 on said other frame and a pixel within said reference patch on said 
reference frame, based on said second patch after the movement and/or 
deformation and on said reference patch; 

first interpolation means for acquiring a first interpolated 
frame whose resolution is higher than each of said frames, by 

3 0 performing interpolation either on the image within said second patch 
of said other frame corresponding to said reference patch or on the 
image within said second patch of said other frame corresponding to 
said reference patch and the image within said reference patch of 
said reference frame such that the pixel values of each pixel therein 

35 are assigned to pixel positions of an integrated image having a 



greater number of pixels than each of said frames , based on employing 
said correspondent relationship; 

second interpolation means for acquiring a second interpolated 
frame whose resolution is higher than each of said frames, by 
5 performing interpolation on the image within said reference patch 
of said reference frame such that the pixel values of each pixel 
therein are assigned to pixel positions of the integrated image ; 

edge information acquisition means for acquiring edge 
information that represents an edge intensity of each pixel within 
10 the image within said reference patch of said reference frame and/or 
image within the patch of said other frame; 

weighting-coefficient acquisition means for acquiring a 
weighting coefficient that makes a weight of said first interpolated 
frame greater as said edge information becomes greater, when 
15 synthesizing said first interpolated frame and second interpolated 
frame, based on said edge information; and 

synthesis means for acquiring a synthesized frame by weighting 
and synthesizing the pixel values of each pixel within said first 
and second interpolated frames, based on said weighting coefficient. 
20 [Claim 6] 

A video image synthesizer comprising: 

sampling means for sampling three or more contiguous frames 
from a video image; 

correspondent relationship estimation means for placing a 

25 reference patch comprising one or a plurality of rectangular areas 
on one of said three or more frames which is used as a reference frame, 
then respectively placing on the others of said three or more frames 
patches which are the same as said reference patch, then moving and/or 
deforming said patches in said other frames so that an image within 

3 0 the patch of each of said other frames coincides with an image within 
said reference patch, and respectively estimating correspondent 
relationships between pixels within the patches of said other frames 
and a pixel within said reference patch of said reference frame , based 
on the patches of said other frames after the movement and/or 

35 deformation and on said reference patch; 



first interpolation means for acquiring a plurality of first 
interpolated frames whose resolution is higher than each of said 
frames, by performing interpolation either on the image within the 
patch. of each of said other frames corresponding to said reference 
5 patch or on the image within the patch of each of said other frames 
corresponding to said reference patch and the image within said 
reference patch of said reference frame, such that the pixel values 
of each pixel therein are assigned to pixel positions of an integrated 
image having a greater number of pixels than each of said frames , 

10 baocd on employing said correspondent relationships; 

second interpolation means for acquiring one or a plurality 
of second interpolated frames whose resolution is higher than each 
of said frames and which are correlated with said plurality of first 
interpolated frames, by performing interpolation on the image within 

15 said reference patch of said reference frame such that the pixel 
values of each pixel therein are assigned to pixel positions of the 
integrated image ; 

edge information acquisition means for acquiring edge 
information that represents an edge intensity of each pixel within 

20 the image within said reference patch of said reference frame and/or 
image within the patch of each of said other frames; 

weighting coefficient acquisition means for acquiring 
weighting coefficients that make a weight of said first interpolated 
frame greater as said edge information becomes greater, when 

25 synthesizing said first interpolated frame and second interpolated 
frame, based on said edge information; and 

synthesis means for acquiring intermediate synthesized frames 
by weighting and synthesizing the pixel values of each pixel within 
said first and second interpolated frames that correspond to each 

3 0 other on the basis of said weighting coefficients, and acquiring a 
synthesized frame by synthesizing said intermediate synthesized 
frames . 
[Claim 7] 

The video image synthesizer as set forth in either one of Claim 
35 5 and Claim 6, wherein when said edge information has been computed 



for each of the pixels that constitute each of said frames, 
said weighting- coef f icient acquisition means performs 
interpolation on said weighting coefficient such that the weighting 
coefficient for each pixel is assigned to pixel positions of the 
5 integrated image , thereby acquiring weighting coefficients for all 
pixels that constitute said first and second interpolated frames. 
[Claim 8] 

The video image synthesizer as set forth in any one of Claim 
5, Claim 6, and Claim 7, wherein said correspondent relationship 

10 estimation means, said first interpolation means, said second 

interpolation means, said edge information acquisition means, said 
weighting-coefficient acquisition means, and said synthesis means 
perform the estimation of said correspondent relationship, 
acquisition of said first interpolated frame, acquisition of said 

15 second interpolated frame, acquisition of said edge information, 
acquisition of said weighting coefficient, and acquisition of said 
synthesized frame, by employing at least one of a brightness component 
and a color component that constitutes said frame . 
[Claim 9] 

20 A program for causing a computer to execute a video image 

synthesis method comprising: 

a procedure of sampling two contiguous frames from a video 

image ; 

a procedure of placing a reference patch comprising one or a 
25 plurality of rectangular areas on one of said two frames which is 
used as a reference frame, then placing on the other of said two frames 
a second patch which is the same as said reference patch, then moving 
and/or deforming said second patch in said other frame so that an 
image within said second patch coincides with an image within said 
3 0 reference patch, and estimating a correspondent relationship between 
a pixel within said second patch on said other frame and a pixel within 
said reference patch on said reference frame, based on said second 
patch after the movement and/or deformation and on said reference 
patch; 

35 a procedure of acquiring a first interpolated frame whose 



resolution is higher than each of said frames, by performing 
interpolation either on the image within said second patch of said 
other frame corresponding to said reference patch or on the image 
within said second patch of said other frame corresponding to said 
5 reference patch and the image within said reference patch of said 
reference frame such that the pixel values of each pixel therein are 
assigned to pixel positions of an integrated image having a greater 
number of pixels than each of said frames , based on employing said 
correspondent relationship; 

10 a procedure of acquiring a second interpolated frame whose 

resolution is higher than each of said frames, by performing 
interpolation on the image within said reference patch of said 
reference frame such that the pixel values of each pixel therein are 
assigned to pixel positions of the integrated image ; 

15 a procedure of acquiring edge information that represents an 

edge intensity of each pixel within the image within said reference 
patch of said reference frame and/or image within the patch of said 
other frame; 

a procedure of acquiring a weighting coefficient that makes 
20 a weight of said first interpolated frame greater as said edge 
information becomes greater, when synthesizing said first 
interpolated frame and second interpolated frame, based on said edge 
information; and 

a procedure of acquiring a synthesized frame by weighting and 
25 synthesizing the pixel values of each pixel within said first and 
second interpolated frames, based on said weighting coefficient. 
[Claim 10] 

A program for causing a computer to execute a video image 
synthesis method comprising: 
3 0 a procedure of sampling three or more contiguous frames from 

a video image; 

a procedure of placing a reference patch comprising one or a 
plurality of rectangular areas on one of said three or more frames 
which is used as a reference frame, then respectively placing on the 
35 others of said three or more frames patches which are the same as 



said reference patch, then moving and/or deforming said patches in 
said other frames so that an image within the patch of each of said 
other frames coincides with an image within said reference patch, 
and respectively estimating correspondent relationships between 
5 pixels within the patches of said other frames and a pixel within 
said reference patch of said reference frame, based on the patches 
of said other frames after the movement and/or deformation and on 
. said reference patch; 

a procedure of acquiring a plurality of first interpolated 

10 frames whose resolution is higher than each of said frames, by 

performing interpolation either on the image within the patch of each 
of said other frames corresponding to said reference patch or on the 
image within the patch of each of said other frames corresponding 
to said reference patch and the image within said reference patch 

15 of said reference frame, such that the pixel values of each pixel 
therein are assigned to pixel positions of an integrated image having 
a greater number of pixels than each of said frames , based on employing 
said correspondent relationships; 

a procedure of acquiring one or a plurality of second 

20 interpolated frames whose resolution is higher than each of said 
frames and which are correlated with said plurality of first 
interpolated frames, by performing interpolation on the image within 
said reference patch of said reference frame such that the pixel 
values of each pixel therein are assigned to pixel positions of the 

25 integrated image ; 

a procedure of acquiring edge information that represents an 
edge intensity of each pixel within the image within said reference 
patch of said reference frame and/ or image within the patch of each 
of said other frames; 

3 0 a procedure of acquiring weighting coefficients that make a 

weight of said first interpolated frame greater as said edge 
information becomes greater, when synthesizing said first 
interpolated frame and second interpolated frame, based on said edge 
information; and 

35 a procedure of acquiring intermediate synthesized frames by 



weighting and synthesizing the pixel values of each pixel within said 
first and second interpolated frames that correspond to each other 
on the basis of said weighting coefficients, and acquiring a 
synthesized frame by synthesizing said intermediate synthesized 
5 frames . 

[Claim 11] 

The program as set forth in either one of Claim 9 and Claim 
10, wherein when said edge information has been computed for each 
of the pixels that constitute each of said frames, 

10 said weighting-coefficient acquisition procedure is a 

procedure of performing interpolation on said weighting coefficient 
such that the weighting coefficient for each pixel is assigned to 
pixel positions of the integrated image and acquiring weighting 
coefficients for all pixels that constitute said first and second 

15 interpolated frames. 
[Claim 12] 

The program as set forth in any one of Claim 9, Claim 
10, and Claim 11, wherein said correspondent relationship estimation 
procedure, said first interpolated frame acquisition procedure, said 

20 second interpolated frame acquisition procedure, said edge 

information acquisition procedure, said weighting-coefficient 
acquisition procedure, and said synthesized- frame acquisition 
procedure are procedures of performing the estimation of said 
correspondent relationship, acquisition of said first interpolated 

25 frame, acquisition of said second interpolated frame, acquisition 
of said edge information, acquisition of said weighting coefficient, 
and acquisition of said synthesized frame by employing at least one 
of a brightness component and a color component that constitutes said 
frame, respectively. 
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1. The inventions defined in the following claims of the present 
application could have been made easily by those having ordinary skill 
in the art, based on the inventions disclosed in the following 

20 publications , which were distributed either domestically or overseas , 
or inventions that were made available to the public via 
telecommunications lines, prior to the filing date of the present 
application. Therefore, patent rights cannot be granted, according 
to the regulations prescribed in Article 29, Paragraph 2 of the Patent 

25 Law. 

NOTES (please refer to the List of Cited References) 

Claims: 1, 3, and 5 

3 0 Cited References: 1 and 2 

"A video image synthesis method. . . [for] acquiring a synthesized 
frame from said predetermined number of frames, based on said 
correspondent relationships" is disclosed in Cited Reference 1. In 
35 addition, Cited Reference 2 discloses determining the number of 



frames to be used for synthesis when synthesizing an image from a 
plurality of frames, based on image characteristics. It would have 
been easy for those skilled in the art to apply the teaching of Cited 
Reference 2 to the invention disclosed in Cited Reference 1. 
5 Accordingly, no inventive step is recognized for the inventions 
defined in Claim 1, 3, and 5 of the present application. 

Claims : 2,4, and 6 

Cited References: 1 through 3 

10 

To cease deriving correspondent relationships with respect to frames 
having correlations which are lower than a predetermined threshold 
value is a well known technique, as disclosed by Cited Reference 2. 
Therefore, no novel technical features are recognized in the 
15 limitations of Claims 2, 4, and 6 of the present application. 

List of Cited References 

1. Y. Nakazawa, T. Komatsu, T. Saito, "Obtainment of Highly Detailed 
20 Digital Images by Frame Integration", Journal of ITE, Vol. 40, 

No. 3 , pp. 299-3 08, The Institute of Image Information and 
Television Engineers, 1995 

2. Japanese Unexamined Patent Publication No. H6-006777 (refer to 
paragraphs [0050] and [0053] through [0055] ) 

25 3. PCT Japanese Publication No. 2002-526227 (refer to paragraph 
[0020] ) 

2. The recitations of the claims of the present application do not 
satisfy the requirements prescribed in Article 36, Paragraph 6, 
3 0 Number 2 of the Patent Law in the following points. 

NOTES 

The recitation "said predetermined number of frames are determined 
35 based on image characteristics of said video image or synthesized 



frame, and said predetermined number of frames are sampled" is unclear 
as to the technical meaning of the image characteristics, as well 
as how the image characteristics are technically used to determine 
the number of frames. Entirely opposite effects may be exhibited 
5 depending on the technical meaning of the above, and therefore the 
inventions cannot be technically specified. 

Accordingly, the inventions defined in Claims 1 through 6 are 
indefinite . 

10 

Record of Prior Art Document Search Results 

• Field of Search: IPC 7 th Edition 

H04N7/00-7/01 

15 • Prior Art Documents 

Japanese Unexamined Patent Publication No. H8-13 0716 
(refer to Figure 1, Figure 3, and the descriptions related 
thereto) 

Japanese Unexamined Patent Publication No. 2001-086508 
20 (refer to Figure 1, Figure 2, Figure 8, and the descriptions 

related thereto) 

Japanese Unexamined Patent Publication No. 2000-354244 
Japanese Unexamined Patent Publication No. 2000-194845 

25 The Record of Prior Art Document Search Results does not constitute 
the Grounds for Rejection. 

If there are any questions regarding the contents of this Notice of 
Grounds for Rejection, or if an interview is desired, please contact 
30 the following. 



Michiharu Sugawara, Patent Examining Division 4 Video Devices 
(Television) 

Tel 03-3581-1101 ext 3580 Fax 03-3501-0715 
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[Claim 1] 

A video image synthesis method comprising the steps of: 
sampling a predetermined number of contiguous frames, which 

include a reference frame and are two or more frames, from said video 

image ; 

placing a reference patch comprising one or a plurality of 
rectangular areas on said reference frame; 

respectively placing patches which are the same as said 
reference patch, on the others of said predetermined number of frames; 

moving and/or deforming said patches in said other frames so 
that an image within the patch of each of said other frames 
approximately coincides with an image within said reference patch; 

respectively acquiring correspondent relationships between 
pixels within the patches of said other frames and a pixel within 
said reference patch of said reference frame, based on the patches 
of said other frames after the movement and/or deformation and on 



said reference patch; and 

acquiring a synthesized frame from said predetermined number 
of frames, based on said correspondent relationships; 

wherein said predetermined number of frames are determined 
5 based on image characteristics the frame rate of said video image 
or synthesized frame such that the predetermined number becomes 
greater as the frame rate is greater, and/ or based on the compression 
rate of each frame of said video image such that the predetermined 
number becomes greater as the compression rate becomes greater, 

10 and/or based on at least one of the pixel size/resolution of said 
video image and a ratio between the pixel size/resolution of said 
synthesized frame and the pixel size/resolution of said video image, 
such that the predetermined number becomes greater as the degree of 
increase in detail from each frame of said video image to said 

15 synthesized frame becomes greater , and said predetermined number of 
frames are sampled. 
[Claim 2] 

The synthesis method as set forth in Claim 1, wherein 
said correspondent relationships are acquired in order of 

2 0 other frames closer to said reference frame, and a correlation is 

acquired between each of said other frames, in which said 
correspondent relationship is acquired, and said reference frame; 

and when said correlation is lower than a predetermined 
threshold value, acquisition of said correspondent relationships is 
25 stopped, and said synthesized frame is obtained based on said 

correspondent relationship by employing said other frames, in which 
said correspondent relationship has been acquired, and said reference 
frame . 
[Claim 3], 

3 0 A video image synthesizer comprising: 

sampling means for sampling a predetermined number of 
contiguous frames, which include a reference frame and are two or 
more frames, from a video image; 

correspondent relationship acquisition means for placing a 
35 reference patch comprising one or a plurality of rectangular areas 



on said reference frame, then respectively placing on the others of 
said predetermined number of frames patches which are the same as 
said reference patch, then moving and/or deforming said patches in 
said other frames so that an image within the patch of each of said 
5 other frames approximately coincides with an image within said 
reference patch, and respectively acquiring correspondent 
relationships between pixels within the patches of said other frames 
and a pixel within said reference patch of said reference frame , based 
on the patches of said other frames after the movement and/or 

10 deformation and on said reference patch; and 

frame synthesis means for acquiring a synthesized frame from 
said predetermined number of frames, based on said correspondent 
relationships acquired by said correspondent relationship 
acquisition means; 

15 wherein said sampling means is equipped with frame -number 

determination means for determining said predetermined number of 
frames on the basis of imago charactcriotica the frame rate of said 
video image or □ynthcaizcd frame such that the predetermined number 
becomes greater as the frame rate is greater, and/or based on the 

2 0 compression rate of each frame of said video image such that the 
predetermined number becomes greater as the compression rate becomes 
greater, and/or based on at least one of the pixel size/resolution 
of said video image and a ratio between the pixel size/resolution 
of said synthesized frame and the pixel size/resolution of said video 

25 image, such that the predetermined number becomes greater as the 
degree of increase in detail from each frame of said video image to 
said synthesized frame becomes greater , and samples said 
predetermined number of frames determined by said frame -number 
determination means. 

30 [Claim 4] 

The video image synthesizer as set forth in Claim 3, wherein 
said correspondent relationship acquisition means acquires 
said correspondent relationships in order of other frames closer to 
said reference frame; 
35 there is provided stoppage means for acquiring a correlation 



between each of said other frames, in which said correspondent 
relationship is acquired by said correspondent relationship 
acquisition means, and said reference frame, and stopping a process 
which is being performed in said correspondent relationship 
5 acquisition means when said correlation is lower than a predetermined 
threshold value; and 

said frame synthesis means acquires said synthesized frame by 
employing said other frames , in which said correspondent relationship 
has been acquired, and said reference frame, based on said 
10 correspondent relationship acquired by said correspondent 
relationship acquisition means. 
[Claim 5] 

A program for causing a computer to execute : 
a sampling process of sampling the a pre determined number of 
15 frames which are contiguous and include a reference frame, from oaid 
a video image , the predetermined number being two or greater ; 

a correspondent relationship acquisition process of placing 
a reference patch comprising one or a plurality of rectangular areas 
on said reference frame, then respectively placing on the others of 
20 said predetermined number of frames patches which are the same as 
said reference patch, then moving and/or deforming said patches in 
said other frames so that an image within the patch of each of said 
other frames approximately coincides with an image within said 
reference patch, and respectively acquiring correspondent 
25 relationships between pixels within the patches of said other frames 
and a pixel within said reference patch of said reference frame, based 
on the patches of said other frames after the movement and/or 
deformation and on said reference patch; and 

a frame synthesis process of acquiring said synthesized frame 
3 0 from said determined number of frames, based on said correspondent 
relationships , wherein: 

the samplingprocess determines the predetermined number based 
on the frame rate of said video image such that the predetermined 
number becomes greater as the frame rate is greater, and/or based 
35 on the compression rate of each frame of said video image such that 



the predetermined number becomes greater as the compression rate 
becomes greater, and/or based on at least one of the pixel 
size/resolution of said video image and a ratio between the pixel 
size/resolution of said synthesized frame and the pixel 
5 size/resolution of said video image, such that the predetermined 
number becomes greater as the degree of increase in detail from each 
frame of said video image to said synthesized frame becomes greater, 
then samples the predetermined number of frames . 
[Claim 6] 

10 The program as set forth in Claim 5, wherein 

said correspondent relationship acquisition process acquires 
said correspondent relationships in order of other frames closer to 
said reference frame; and 

said computer is further caused to execute a process of 

15 acquiring a correlation between each of said other frames, in 
which said correspondent relationship is acquired, and said 
reference frame, and stopping said correspondent relationship 
acquisition process when said correlation is lower than a 
predetermined threshold value. 
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In the case that the applicant has comments regarding the rejection, 
please file an Argument within 60 days of the issue date of this 
Notice. 

15 GROUNDS 

GROUND A 

The recitations of the claims and the Detailed Description of the 

2 0 Drawings of the present application do not satisfy the requirements 

prescribed in Article 36, Paragraph 4, Number 1, and Article 36, 
Paragraph 6, Number 1 of the Patent Law in the following points. 

GROUND B 

25 

The recitations of the claims of the present application do not 
satisfy the requirements prescribed in Article 36, Paragraph 6, 
Number 2 of the Patent Law in the following points. 

3 0 GROUND C 

The inventions defined in the following claims of the present 
application could have been made easily by those having ordinary skill 
in the art, based on the inventions disclosed in the following 
35 publications, which were distributed either domestically or overseas, 



or inventions that were made available to the public via 
telecommunications lines, prior to the filing date of the present 
application. Therefore, patent rights cannot be granted, according 
to the regulations prescribed in Article 29, Paragraph 2 of the Patent 
5 Law. 

NOTES (please refer to the List of Cited References) 

1 . Ground A 
10 • Regarding Claims 1, 8, and 15 

• Remarks 

Claims 1, 8, and 15 recite "obtaining a contiguous frame group by 
detecting a plurality of frames that represent contiguous scenes in 

15 a video image ; placing a reference patch comprising one or a plurality 
of rectangular areas on one of said plurality of frames included in 
said contiguous frame group which is used as a reference frame" . 
However, there is no specific description in the Detailed Description 
of the Invention or the Figures regarding how a starting frame is 

20 determined, nor regarding how the "reference frame" is determined, 
when "detecting a plurality of frames" . 

Accordingly, the inventions defined in Claims 1, 8, and 15 are not 
described in the Detailed Description of the Invention, and the 
25 Detailed Description of the Invention of the present application does 
not describe the inventions defined in Claims 1, 8, and 15 clearly 
and sufficiently to a degree that would enable those skilled in the 
art to carry them out . 

3 0 • Regarding Claims 2 and 9 

• Remarks 

Claims 2 and 9 recite "started from said reference frame" . However, 
there is no specific description in the Detailed Description of the 
35 Invention or the Figures regarding how the "reference frame" is 



determined. 

In addition, Claims 2 and 9 recite "between a pair of said adjacent 
frames in which said correlation is lower than a predetermined first 
5 threshold value" . However, there is no specific description in the 
Detailed Description of the Invention or the Figures regarding what 
value described in the specification the "first threshold value" 
corresponds to. 

10 Accordingly, the inventions defined in Claims 2 and 9 are not 

described in the Detailed Description of the Invention, and the 
Detailed Description of the Invention of the present application does 
not describe the inventions defined in Claims 2 and 9 clearly and 
sufficiently to a degree that would enable those skilled in the art 

15 to carry them out . 

• .Regarding Claims 3 and 10 

• Remarks 

2 0 Claims 3 and 10 recite "the sum of the Euclidean distances for said 

three components is computed; and when said sum is a predetermined 
second threshold value or greater, the correlation between said 
adjacent frames is lower than said predetermined first threshold 
value" . However, there is no specific description in the Detailed 
25 Description of the Invention or the Figures regarding what value 
described in the specification the "second threshold value" 
corresponds to. 

In addition, there is no specific description in the Detailed 

3 0 Description of the Invention or the Figures regarding "the sum of 

the Euclidean distances for said three components is computed; and 
when said sum is a predetermined second threshold value or greater, 
the correlation between said adjacent frames is lower than said 
predetermined first threshold value" . 

35 



Accordingly, the inventions defined in Claims 3 and 10 are not 
described in the Detailed Description of the Invention, and the 
Detailed Description of the Invention of the present application does 
not describe the inventions defined in Claims 3 and 10 clearly and 
5 sufficiently to a degree that would enable those skilled in the art 
to carry them out . 

• Regarding Claims 4 and 11 

• Remarks 

10 

Claims 4 and 11 recite "said histogram is computed by dividing each 
component among said three components by a value greater than 1" . 
However, there is no specific description in the Detailed Description 
of the Invention or the Figures regarding what value described in 
15 the specification the "value greater than 1" corresponds to, nor is 
there a description regarding why there is no upper limit to the value . 

In addition, there is no specific description in the Detailed 
Description of the Invention or the Figures regarding "said histogram 

2 0 is computed by dividing each component among said three components 

by a value greater than 1" . 

Accordingly, the inventions defined in Claims 4 and 11 are not 
described in the Detailed Description of the Invention, and the 
25 Detailed Description of the Invention of the present application does 
not describe the inventions defined in Claims 4 and 11 clearly and 
sufficiently to a degree that would enable those skilled in the art 
to carry them out . 

3 0 • Regarding Claims 5 and 12 

• Remarks 

Claims 5 and 12 recite "the sum of absolute values of the differences 
for all corresponding pixels is computed; and when said sum is a third 
35 threshold value or greater, said correlation between adj acent frames 



is lower than said predetermined first threshold value" . However, 
there is no specific description in the Detailed Description of the 
Invention or the Figures regarding what value described in the 
specification the "third threshold value" corresponds to. 

5 

In addition, there is no specific description in the 
Detailed Description of the Invention or the Figures regarding "the 
sum of absolute values of the differences for all corresponding pixels 
is computed; and when said sum is a third threshold value or greater, 
10 said correlation between adjacent frames is lower than said 
predetermined first threshold value" . 

Accordingly, the inventions defined in Claims 5 and 12 are not 
described in the Detailed Description of the Invention, and the 
15 Detailed Description of the Invention of the present application does 
not describe the inventions defined in Claims 5 and 12 clearly and 
sufficiently to a degree that would enable those skilled in the art 
to carry them out . 

20 2 . Ground B 

• Regarding Claims 1 through 5, 8 through 12, and 15 

• Remarks 

As pointed out in Ground A above, the inventions defined in Claims 
25 1 through 5, 8 through 12, and 15 are not described in the Detailed 
Description of the Invention. Therefore, they are indefinite. 

3 . Ground C 

• Regarding Claims 1 through 15, Cited Reference 1 
3 0 • Remarks 

Examination regarding inventive step was performed, disregarding the 
issues pointed out in Grounds A and B. Cited Reference 1 discloses 
providing four sided patches in a reference frame and other frames, 
35 obtaining correlative relationships among images within the four 



sided patches, then synthesizing the reference frame and the other 
frames . 

If new Grounds for Rejection are found, the applicant will be 
notified. 

List of Cited References 

1. Y. Nakazawa, T. Komatsu, T. Saito, "(Detainment of Highly Detailed 
Digital Images by Frame Integration", Journal of ITE, Vol. 40, 
No. 3 , pp. 299-308, The Institute of Image Information and 
Television Engineers, 1995 

Record of Prior Art Document Search Results 

• Field of Search: IPC H04N 5/262 - 5/28 

G06T 1/00 - 5/50 

• Prior Art Documents 

Japanese Unexamined Patent Publication No. 2000-187731 
Japanese Unexamined Patent Publication No. 2001-134761 
Japanese Unexamined Patent Publication No. 2001-189889 

The Record of Prior Art Document Search Results does not constitute 
the Grounds for Rejection. 

If there are any questions regarding the contents of this Notice of 
Grounds for Rejection, or if an interview is desired, please contact 
the following. 



Jun Yoshikawa, Patent Examining Division 4 Video Devices (Television) 
Tel 03-3581-1101 ext 3539 Fax 03-3501-0715 
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File No. 



Application No. 2002-284127 



Submitted: 6/25/2007 



Amendment 

Commissioner of the Patent Office 



[Name of Document] 
[Addressee] 
[Indication of Case] 

[Application Number] 
[Applicant] 

[ID Number] 

[Name] 
[Representative] 

[ID Number] 

[Patent Attorney] 

[Name] 
[Submission Number] 
[Amendment 1 ] 

[Document to be Amended] 

[Item to be Amended] 

[Manner of Amendment] 

[Contents of Amendment] 
[Scope of Demand for Patent] 
[Claim 1] 

A video image synthesis method comprising the steps of: 
obtaining a contiguous frame group including a predetermined 
reference frame by detecting a plurality of frames in the vicinity 
of the predetermined reference frame within a video image and of which 
the characteristics of the contents thereof are similar that 
3 in a video image ; 



2002-284127 

306037311 
FUJIFILM Corp. 

100073184 

Masashi Yanagida 
198808 

Specification 

Scope of Demand for Patent 

Change 



rcprcacnt contiguous □ccnco i 

placing a reference patch comprising one or a plurality of 
rectangular areas on the reference frame one of 3aid plurality of 
frames included in said contiguous frame group which is used ao a 
reference frame ; 

respectively placing patches which are the same as said 
reference patch, on the others of said plurality of frames; 

moving and/or deforming said patches in said other frames so 
that an image within the patch of each of said other frames 
approximately coincides with an image within said reference patch; 



respectively acquiring correspondent relationships between 
pixels within the patches of said other frames and a pixel within 
said reference patch of said reference frame, based on the patches 
of said other frames after the movement and/or deformation and on 
5 said reference patch; and 

acquiring a synthesized frame from said plurality of frames, 
based on said correspondent relationships . 
[Claim 2] 

The synthesis method as set forth in Claim 1, wherein 
10 a correlation between adjacent frames, which is started from 

said reference frame, is acquired; and 

said contiguous frame group that is detected comprises frames 
ranging from said reference frame to a frame, which is closer to said 
reference frame, between a pair of said adjacent frames in which said 
15 correlation is lower than a predetermined first threshold value. 
[Claim 3] 

The synthesis method as set forth in Claim 2, wherein 

a histogram is computed for at least one of the YCC components 
of each of said adjacent frames; 
20 a Euclidean distance for each component between said adjacent 

frames is computed by employing said histogram; 

the sum of the Euclidean distances for said three components 
is computed; and 

when said sum is a predetermined second threshold value or 
25 greater, the correlation between said adjacent frames is lower than 
said predetermined first threshold value. 
[Claim 4] 

The synthesis method as set forth in Claim 2, wherein 
a difference between pixel values of corresponding pixels of 
3 0 said adjacent frames is computed for all corresponding pixels; 

the sum of absolute values of the differences for all 
corresponding pixels is computed; and 

when said sum is a third threshold value or greater, said 
correlation between adjacent frames is lower than said predetermined 
3 5 first threshold value. 



[Claim 5] 

The synthesis method as set forth in any one of Claims 2 through 

4 , wherein said correlation is computed by employing a reduced image 
or thinned image of each frame. 

5 [Claim 6] 

The synthesis method as set forth in any one of Claims 1 through 

5, wherein the detection of frames that constitute said contiguous 
frame group is stopped when the number of detected frames reaches 
a predetermined upper limit value. 

10 [Claim 7] 

A video image synthesizer comprising: 

contiguous frame group detection means for obtaining a 
contiguous frame group including a predetermined reference frame by 
detecting a plurality of frames in the vicinity of the predetermined 

15 reference frame within a video image and of which the characteristics 
of the contents thereof are similar that rcprcacnt contiguous □ccnca 
in a video image ; 

correspondent relationship acquisition means for placing a 
reference patch comprising one or a plurality of rectangular areas 

20 on the reference frame one of said plurality of frames included in 
said contiguous frame group which io used as a reference frame , then 
respectively placing on the others of said plurality of frames patches 
which are the same as said reference patch, then moving and/or 
deforming said patches in said other frames so that an image within 

25 the patch of each of said other frames approximately coincides with 
an image within said reference patch, and respectively acquiring 
correspondent relationships between pixels within the patches of said 
other frames and a pixel within said reference patch of said reference 
frame, based on the patches of said other frames after the movement 

3 0 and/or deformation and on said reference patch; and 

frame synthesis means for acquiring a synthesized frame from 
said plurality of frames, based on said correspondent relationships 
acquired by said correspondent relationship acquisition means. 
[Claim 8] 

35 The video image synthesizer as set forth in Claim U, wherein 



said contiguous frame group detection means is equipped with 
correlation computation means for computing a correlation between 
adjacent frames which is started from said reference frame; and 

said contiguous frame group, which is detected by said 
5 contiguous frame group detection means , comprises frames ranging from 
said reference frame to a frame, which is closer to said reference 
frame, between a pair of said adjacent frames in which said 
correlation is lower than a predetermined first threshold value. 
[Claim 9] 

10 The video image synthesizer as set forth in Claim 8, wherein 

said correlation computation means computes a histogram for 
at least one of the YCC components of each of said adjacent frames, 
also computes a Euclidean distance for each component between said 
adjacent frames by employing said histogram, and computes the sum 

15 of the Euclidean distances for said three components; and 

when said sum is a predetermined second threshold value or 
greater, said contiguous frame group detection means judges that said 
correlation between said adjacent frames is lower than said 
predetermined first threshold value. 

20 [Claim 10] 

The video image synthesizer as set forth in Claim 8, wherein 
said correlation computation means computes a difference 
between pixel values of corresponding pixels of said adjacent frames 
and also computes the sum of absolute values of the differences for 
25 all corresponding pixels; and 

when said sum is a third threshold value or greater, said 
contiguous frame group detection means judges that said correlation 
between adjacent frames is lower than said predetermined first 
threshold value. 
3 0 [Claim 11 ] 

The video image synthesizer as set forth in any one of Claims 
8 through 10, wherein said correlation computation means computes 
said correlation, employing a reduced image or thinned image of each 
frame . 
35 [Claim 12] 



The video image synthesizer as set forth in any one of Claims 
7 through 11 , further comprising stoppage means for stopping the 
detection of frames, which constitute said contiguous frame group, 
when the number of frames detected by said contiguous frame group 
5 detection means reaches a predetermined upper limit value. 

69. A program for causing a computer to execute: 
a contiguous frame group detection process of obtaining a 
contiguous frame group including a predetermined reference frame by 
detecting a plurality of frames in the vicinity of the predetermined 
10 reference frame within a video image and of which the characteristics 
of the contents thereof are similar that represent contiguous scenes 
in a video image ; 

a correspondent relationship acquisition process of placing 
a reference patch comprising one or a plurality of rectangular areas 
15 on the reference frame one of aaid plurality of framco included in 
said contiguous frame group which io uaod ao a reference frame , then 
respectively placing on the others of said plurality of frames patches 
which are the same as said reference patch, then moving and/or 
deforming said patches in said other frames so that an image within 
20 the patch of each of said other frames approximately coincides with 
an image within said reference patch, and respectively acquiring 
correspondent relationships between pixels within the patches of said 
other frames and a pixel within said reference patch of said reference 
frame, based on the patches of said other frames after the movement 
25 and/or deformation and on said reference patch; and 

a frame synthesis process of acquiring a synthesized frame from 
said plurality of frames, based on said correspondent relationships . 
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NOTICE OF DECISION OF REJECTION 



Patent Application Number: 
Drafting Date: 
Patent Office Examiner: 
Title of Invention: 

Applicant : 

Applicant ' s Representative : 



2002-284127 

October 4, 2007 

Jun Yoshikawa 9651 5P00 

Method, Apparatus, and Program for 

Video Synthesis 

FUJIFILM Corp. 

Masashi Yanagida et al . 



The present application is to be rejected based on Grounds A 
through C described in the Notice of Grounds for Rejection, dated 
April 19, 2007. 

15 Note that the contents of the Argument and the Amendment were 

considered, but no basis sufficient to overturn the Grounds for 
Rejection was found. 



Remarks 

20 

In the Argument, the Applicant asserts that paragraph [0006] of the 
original specification describes the manner in which the reference 
frame is determined. However, paragraph [0006] neither discloses 
nor suggests the manner in which the reference frame is determined 
25 (the same applies to paragraph [0028] ) . 



There is no basis for the above assertion, and therefore Grounds A 
and B still remain unresolved. 



3 0 Regarding Ground C, disregarding the points mentioned in Grounds A 
and B, no significant difference is recognized between the contents 
of the present application and the contents of Cited Reference 1. 
Accordingly, it is recognized that the inventions of the present 
application would have been easily invented by those skilled in the 

35 art, based on the contents of Cited Reference 1. 



If the applicant is dissatisfied with this decision, an appeal may- 
be filed with the Commissioner of the Patent Office within 3 0 days 
5 (90 days for foreign residents) of delivery of a copy of this decision 
(Article 121, Paragraph 1 of the Patent Law) . 

(Teaching in accordance with Article 46, Paragraph 2 of the 
Administrative Case Litigation Law) 
10 It is not possible to file a suit to revoke this decision without 
filing the aforementioned appeal. In the case that the results of 
the aforementioned appeal are not satisfactory, then a suit may be 
filed to revoke the appeal decision (Article 178, Paragraph 6 of the 
Patent Law) . 

15 
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File No. 

[Name of Document] 
[Addressee] 
[Indication of Case] 

[Appeal Number] 

[Application Number] 
[Applicant] 

[ID Number] 

[Name] 
[Representative] 

[ID Number] 

[Patent Attorney] 

[Name] 
[Amendment 1 ] 

[Document to be Amended] 

[Item to be Amended] 

[Manner of Amendment] 

[Contents of Amendment] 
[Scope of Demand for Patent] 
[Claim 1] 

A video image synthesis method comprising the steps of: 
obtaining a contiguous frame group including a n arbitrarily 
set predetermined reference frame and at least portion of frames in 
the vicinity thereof, by calculating correlations in image data among 
the predetermined reference frame and at least a portion of frames 
in the vicinity thereof, then detecting a plurality of frames in the 
vicinity of the predetermined reference frame within a video image 
aad of which the characteristics of the contents thereof are similar^ 
based on the correlations ; 

placing a reference patch comprising one or a plurality of 
rectangular areas on the reference frame included in said contiguous 
frame group; 

respectively placing patches which are the same as said 
reference patch, on the others of said plurality of frames; 

moving and/or deforming said patches in said other frames so 



that an image within the patch of each of said other frames 
approximately coincides with an image within said reference patch; 

respectively acquiring correspondent relationships between 
pixels within the patches of said other frames and a pixel within 
5 said reference patch of said reference frame, based on the patches 
of said other frames after the movement and/or deformation and on 
said reference patch; and 

acquiring a synthesized frame from said plurality of frames, 
based on said correspondent relationships. 
10 [Claim 2] 

The synthesis method as set forth in Claim 1, wherein 

a correlation between adjacent frames, which is started from 
said reference frame, is acquired; and 

said contiguous frame group that is detected comprises frames 
15 ranging from said reference frame to a frame, which is closer to said 
reference frame, between a pair of said adjacent frames in which said 
correlation is lower than a predetermined first threshold value. 
[Claim 3] 

The synthesis method as set forth in Claim 2, wherein 
20 a histogram is computed for at least one of the YCC components 

of each of said adjacent frames; 

a Euclidean distance for each component between said adjacent 
frames is computed by employing said histogram; 

the sum of the Euclidean distances for said three components 
25 is computed; and 

when said sum is a predetermined second threshold value or 
greater, the correlation between said adjacent frames is lower than 
said predetermined first threshold value. 
[Claim 4] 

3 0 The synthesis method as set forth in Claim 2, wherein 

a difference between pixel values of corresponding pixels of 
said adjacent frames is computed for all corresponding pixels; 

the sum of absolute values of the differences for all 
corresponding pixels is computed; and 
35 when said sum is a third threshold value or greater, said 



correlation between adjacent frames is lower than said predetermined 
first threshold value. 
[Claim 5] 

The synthesis method as set forth in any one of Claims 2 through 
5 4, wherein said correlation is computed by employing a reduced image 
or thinned image of each frame. 
[Claim 6] 

The synthesis method as set forth in any one of Claims 1 through 
5, wherein the detection of frames that constitute said contiguous 
10 frame group is stopped when the number of detected frames reaches 
a predetermined upper limit value. 
[Claim 7] 

A video image synthesizer comprising: 

contiguous frame group detection means for obtaining a 
15 contiguous frame group including a n arbitrarily set predetermined 
reference frame and at least portion of frames in the vicinity thereof, 
by calculating correlations in image data among the predetermined 
reference frame and at least a portion of frames in the vicinity 
thereof, then detecting a plurality of frames in the vicinity of the 
20 predetermined reference frame within a video imago and of which the 
characteristics of the contents thereof are similar , based on the 
correlations ; 

correspondent relationship acquisition means for placing a 
reference patch comprising one or a plurality of rectangular areas 

25 on the reference frame included in said contiguous frame group, then 
respectively placing on the others of said plurality of frames patches 
which are the same as said reference patch, then moving and/or 
deforming said patches in said other frames so that an image within 
the patch of each of said other frames approximately coincides with 

3 0 an image within said reference patch, and respectively acquiring 
correspondent relationships between pixels within the patches of said 
other frames and a pixel within said reference patch of said reference 
frame, based on the patches of said other frames after the movement 
and/or deformation and on said reference patch; and 

35 frame synthesis means for acquiring a synthesized frame from 



said plurality of frames, based on said correspondent relationships 
acquired by said correspondent relationship acquisition means. 
[Claim 8] 

The video image synthesizer as set forth in Claim U, wherein 
5 said contiguous frame group detection means is equipped with 

correlation computation means for computing a correlation between 
adjacent frames which is started from said reference frame; and 

said contiguous frame group, which is detected by said 
contiguous frame group detection means, comprises frames ranging from 
10 said reference frame to a frame, which is closer to said reference 
frame, between a pair of said adjacent frames in which said 
correlation is lower than a predetermined first threshold value. 
[Claim 9] 

The video image synthesizer as set forth in Claim 8, wherein 
15 said correlation computation means computes a histogram for 

at least one of the Y, Cb, and Cr components of each of said adjacent 
frames (where the Y component is a luminance component and the Cb 
and Cr components are color difference components) , also computes 
a Euclidean distance for each component between said adjacent frames 
20 by employing said histogram, and computes the sum of the Euclidean 
distances for said three components; and 

when said sum is a predetermined second threshold value or 
greater, said contiguous frame group detection means judges that said 
correlation between said adjacent frames is lower than said 
25 predetermined first threshold value. 
[Claim 10] 

The video image synthesizer as set forth in Claim 8, wherein 
said correlation computation means computes a difference 
between pixel values of corresponding pixels of said adjacent frames 
3 0 and also computes the sum of absolute values of the differences for 
all corresponding pixels; and 

when said sum is a third threshold value or greater, said 
contiguous frame group detection means judges that said correlation 
between adjacent frames is lower than said predetermined first 
35 threshold value. 



[Claim 11] 

The video image synthesizer as set forth in any one of Claims 
8 through 10, wherein said correlation computation means computes 
said correlation, employing a reduced image or thinned image of each 
5 frame . 

[Claim 12] 

The video image synthesizer as set forth in any one of Claims 
7 through 11, further comprising stoppage means for stopping the 
detection of frames, which constitute said contiguous frame group, 
10 when the number of frames detected by said contiguous frame group 
detection means reaches a predetermined upper limit value. 
69. A program for causing a computer to execute: 
a contiguous frame group detection process of obtaining a 
contiguous frame group including a n arbitrarily set predetermined 
15 reference frame and at least portion of frames in the vicinity thereof , 
by calculating correlations in image data among the predetermined 
reference frame and at least a portion of frames in the vicinity 
thereof, then detecting a plurality of frames in the vicinity of the 
prcdotcrmincd reference frame within a video image and of which the 

2 0 characteristics of the contents thereof are similar , based on the 

correlations ; 

a correspondent relationship acquisition process of placing 
a reference patch comprising one or a plurality of rectangular areas 
on the reference frame included in said contiguous frame group, then 
25 respectively placing on the others of said plurality of frames patches 
which are the same as said reference patch, then moving and/or 
deforming said patches in said other frames so that an image within 
the patch of each of said other frames approximately coincides with 
an image within said reference patch, and respectively acquiring 

3 0 correspondent relationships between pixels within the patches of said 

other frames and a pixel within said reference patch of said reference 
frame, based on the patches of said other frames after the movement 
and/or deformation and on said reference patch; and 

a frame synthesis process of acquiring a synthesized frame from 
35 said plurality of frames, based on said correspondent relationships. 
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Application Number: 2002-284128 

Drafting Date: April 19, 2007 

Examiner: Jun Yos hikawa 9651 5P00 

Applicant's Representative: Masashi Yanagida et al . 

Applied Statutes: Article 29, Paragraph 2 

The present application is rejected based on the following Grounds. 
In the case that the applicant has comments regarding the rejection, 
please file an Argument within 60 days of the issue date of this 
Notice . 

GROUNDS 

The inventions defined in the following claims of the present 
application could have been made easily by those having ordinary skill 
in the art, based on the inventions disclosed in the following 
publications, which were distributed either domestically or overseas, 
or inventions that were made available to the public via 
telecommunications lines, prior to the filing date of the present 
application. Therefore, patent rights cannot be granted, according 
to the regulations prescribed in Article 29, Paragraph 2 of the Patent 



Law. 



NOTES (please refer to the List of Cited References) 

• Regarding Claims 1 through 11, Cited References 1 and 2 

• Remarks 

Cited Reference 1 discloses providing four sided patches in a 
reference frame and other frames, obtaining correlative 
relationships among images within the four sided patches, then 
synthesizing the reference frame and the other frames. 



jgg##:P27187T 3§j£#-S-: 198807 5Si£ B : SPb!c19*E 4J3 24 B 1 



mm $ 



#M 2002-284128 



9651 5P00 



mm 29^2 ± * wf**^ s^i ^x-t & v\ 

la (§iffl5:iit^fcov>-cfi5I^S:i^-Jl#R§) 
• 3lffiS:tti, 2 



Wtr-^g^^f, 1 9 9 5^ 3^200, *4 9*»3*, 2 9 9-3 0 81 
, ISSN 0386-6831 

2 . %?mw io-2 8 5 5 8 i 



$\ % x m m - n 



Hg#-§-:P27187T 3§ 198807 5Bi*B :¥fifel9*E 4J3 24B 2/E 



•W54Lfc#» IPC H04N 5/262 - 5/28 
G06T 1/00 - 5/50 

■ MX! #12 0 0 0-1 8 7 7 3 1 
#12001-134761 
M 2001-189889 



#fF#2HM mmm (t v tr v 5 a is) 

TEL. 03 (3581) 1101 EXT. 3 5 39 
FAX. 03 (3501) 0715 



File Number: P27186J Issue Number: 516505 Issue Date: 10/16/2007 



NOTICE OF DECISION OF REJECTION 



Patent Application Number: 
5 Drafting Date: 

Patent Office Examiner: 
Title of Invention: 

Applicant : 
10 Applicant's Representative: 



2002-284128 

October 4, 2 007 

Jun Yoshikawa 9651 5P00 

Method, Apparatus, and Program for 

Video Synthesis 

FUJIFILM Corp. 

Masashi Yanagida et al . 



The present application is to be rejected based on Grounds A 
through C described in the Notice of Grounds for Rejection, dated 
April 19, 2007. 

15 Note that the contents of the Argument and the Amendment were 

considered, but no basis sufficient to overturn the Grounds for 
Rejection was found. 

Remarks 

20 

In the Argument, the Applicant asserts that Cited Reference 2 detects 
a moving body within a single frame (image) based on a plurality of 
frames (images) , which is different from the invention of the present 
application, which detects a frame group (image group) that 
25 constitutes an important scene from a plurality of frames (plurality 
of images) within a video image. However, Cited Reference 2 is 
directed to an automatic surveillance system, and therefore it is 
recognized that image groups that constitute important scenes are 
extracted. 

30 

Accordingly, it is recognized that the inventions of the present 
application would have been easily invented by those skilled in the 
art, based on the contents of Cited Reference 1 and Cited Reference 
2 . 

35 



If the applicant is dissatisfied with this decision, an appeal may- 
be filed with the Commissioner of the Patent Office within 3 0 days 
(90 days for foreign residents) of delivery of a copy of this decision 
5 (Article 121, Paragraph 1 of the Patent Law) . 

(Teaching in accordance with Article 46, Paragraph 2 of the 
Administrative Case Litigation Law) 

It is not possible to file a suit to revoke this decision without 
10 filing the aforementioned appeal. In the case that the results of 
the aforementioned appeal are not satisfactory, then a suit may be 
filed to revoke the appeal decision (Article 178, Paragraph 6 of the 
Patent Law) . 

15 



20 



25 



I hereby certify that the above contents are identical to those 
recorded in the file. 

Date of Certification: October 11, 2007 
35 Emiko Hirase, Officer, Ministry of Business and Industry 
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